3 - Architecting A Vcloud 1.6 - Technical White Paper PDF

VMware vCloud
Architecting
a vCloud
Version 1.6
T e c h n i c a l W HI T E P A P E R
VMware vCloud
Architecting a vCloud
Table of Contents
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1. What is a VMware vCloud?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.1 Document Purpose and Assumptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Cloud Computing and vCloud Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 vCloud Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 vCloud Infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4.1 vCloud Management Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.2 Compute Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.3 Storage Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.4 Networking Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.5 Component Placement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.6 vCloud Consumer Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4.7 vCloud Logical Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2. vCloud Director Constructs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3. vCloud Consumer Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1 Cloud Consumer Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2 Establish Provider Virtual Datacenters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.1 Public Cloud Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.2 Private Cloud Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.3 Provider Virtual Datacenter Special Use Cases . . . . . . . . . . . . . . . . . . . . . 18
3.2.4 Compute Resources Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.2.5 Storage Resources Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2.6 Networking Resources Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3 Multi-Site/Multi-Geo Clouds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3.1 Scenario #1Common User Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3.2 Scenario #2Common Set of Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . .22
3.3.3 Suggested Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.3.4 Other Multi-Site Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3.5 Merging Chargeback Reports. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3.6 Synchronizing Catalogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
T ECHNICAL W HI T E P A P E R / 2
VMware vCloud
4. Providing Cloud Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1 Establish Organizations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.1.1 Administrative Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.1.2 Standard Organizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.2 Establish Networking Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.2.1 External Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2.2 Network Pools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2.3 Cisco Nexus 1000V Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.3 Establish Networking OptionsPublic vCloud Example . . . . . . . . . . . . . . . . . . . . 26
4.3.2 Network Pools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3.3 Organization Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.4 Establish Networking OptionsPrivate vCloud Example. . . . . . . . . . . . . . . . . . . . 30
4.4.2 Network Pools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.4.3 Organization Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.5 Establish Organization Virtual Datacenters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.5.1 Public vCloud Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.5.2 Private vCloud Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.6 Create vApp Templates and Media Catalogs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.6.1 Auto-Joining Active Directory Domains. . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.6.2 Establish Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.6.3 Accessing your vCloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.6.4 Deploy vApps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.6.5 Employ Chargeback or Showback. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5. Extending vCloud Capabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.1 Core vCloud Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.2 vCloud Request Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.3 vCloud API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.4 vCenter Orchestrator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.4.1 Cloud Administration Orchestration Examples. . . . . . . . . . . . . . . . . . . . . . 37
5.4.2 Organization Administration Orchestration Examples. . . . . . . . . . . . . . . 37
5.4.3 Cloud Consumer Operation Orchestration Examples. . . . . . . . . . . . . . . . 37
VMware vCloud
5.5 vCloud Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

5.5.1 vCloud Connector Placement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.5.2 vCloud Connector Example Usage Scenarios. . . . . . . . . . . . . . . . . . . . . . . 39
5.5.3 vCloud Connector Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6. Managing the vCloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.1 Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.1.1 Management Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.1.2 Cloud Consumer Resources and Workloads. . . . . . . . . . . . . . . . . . . . . . . . 40
6.2 Logging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.2.1 Logging Architectural Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.2.2 Logging as a Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.3 End-to-End Security Considerations with vCloud. . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.3.1 vCloud Environment Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.3.2 User Access Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.3.3 Securing Workloads at the NetworkLevel Workload Security . . . . . . 43
6.4 Workload Availability Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.4.1 Uptime SLAs at 99.99%. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.4.2 Load Balancing of vCloud Director Cells . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.4.3 I/O Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6.4.4 Disaster Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6.4.5 Backup and Restore of vApps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
7. Sizing the vCloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7.1 Initial Sizing of Cloud Consumer Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7.2 Capacity Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
8. Implementing Your vCloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
9. Appendix: vCloud Director Cell Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
10. Appendix: vCloud Availability Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
11. Appendix: Security Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
11.1 Network Access Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
11.2 Compliance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61
11.3 Use Cases: Why Logs Should be Available . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
11.3.1 Example Compliance Use Cases for Logs . . . . . . . . . . . . . . . . . . . . . . . . . . 64
11.3.2 VMware vCloud Log Sources for Compliance. . . . . . . . . . . . . . . . . . . . . . . 65
11.4 vCloud Director Diagnostic and Audit Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
11.5 Load Balancer Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
12. Appendix: Signed Certificates with vCloud Director. . . . . . . . . . . . . . . . . . . . . . . . . . . 70
VMware vCloud
13. Appendix: Capacity Planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

13.1 Cloud Administrator (Service Provider) Perspective. . . . . . . . . . . . . . . . . . . . . . . . 87
13.2 Network Capacity Planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
14. Appendix: Capacity Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
14.1 vCloud-Specific Capacity Forecasting (Demand Management) . . . . . . . . . . . . . 93
14.2 Capacity Monitoring and Establishing Triggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
14.3 Capacity Management Manual ProcessesProvider Virtual Datacenter. . . . . . 94
14.4 End-Customer (Organization) Administrator Perspective. . . . . . . . . . . . . . . . . . . 95
14.5 Organization Virtual DatacenterSpecific Capacity Forecasting . . . . . . . . . . . . 97
14.6 Capacity Management Manual ProcessesOrganization Virtual Datacenter. . . 100
VMware vCloud
List of Figures
Figure 1. vCloud Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Figure 2. Core vCloud Logical Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Figure 3. vCloud Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Figure 4. vCloud Logical Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Figure 5. vCloud Director Construct to vSphere Mapping . . . . . . . . . . . . . . . . . . . . . . . . . 15
Figure 6. vCloud Consumer Resource Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Figure 7. Two Sites with Local vCloud Director Instances Managing Local vCenters . . 21
Figure 8. Remote Console Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Figure 9. Two Sites with Isolated vCloud Director Instances. . . . . . . . . . . . . . . . . . . . . . . . 23
Figure 10. Example Diagram of Provider Networking for a Public vCloud . . . . . . . . . . . 27
Figure 11. Configure External IPs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Figure 12. vCloud Director Logical Networking w/ Cisco Nexus 1000V . . . . . . . . . . . . . 29
Figure 13. Example Diagram of Provider Networking for a Private vCloud. . . . . . . . . . . 30
Figure 14. vCloud Connector Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Figure 15. Architectural Example Drawing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Figure 16. Configure Firewall Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Figure 17. Reference Architecture Kit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Figure 18. Log Collection in the Cloud Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Figure 19. Architecture of vCloud Components and Log Collection. . . . . . . . . . . . . . . . . 65
Figure 20. Infrastructure Layers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
VMware vCloud
List of Tables
Table 1. Reference Documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Table 2. vCloud Components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Table 3. vCloud Director Constructs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Table 4. Component Requirements for a Management Cluster . . . . . . . . . . . . . . . . . . . . . 19
Table 5. Network Pool Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Table 6. vCloud vApp Requirements Checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Table 7. Definition of Resource Pool and Virtual Machine Split. . . . . . . . . . . . . . . . . . . . . . 49
Table 8. Memory, CPU, Storage, and Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Table 9. Example Consolidation Ratios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Table 10. MBeans Used To Monitor vCloud Cells. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Table 11. vCloud Availability Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Table 12. Network Access Security Use Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Table 13. Audit Concerns Within The Cloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Table 14. vCloud Component Logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Table 15. Other Component Logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Table 16. Load Balancer Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Table 17. Certificate Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Table 18. vSphere Host Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Table 19. Determing Redundancy Overhead. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Table 20. Network Capacity Planning Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Table 21. Capacity Monitoring Metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Table 22. Organization Virtual Datacenter Units of Consumption. . . . . . . . . . . . . . . . . . . 95
Table 23. Recommended Organization Virtual Datacenter Capacity Thresholds. . . . . . 95
Table 24. Sample Organization Virtual Datacenter Resource Allocation . . . . . . . . . . . . . 96
Table 25. Organization Virtual Datacenter Trending Information. . . . . . . . . . . . . . . . . . . . 97
Table 26. Organization Virtual Datacenter Capacity Trending Variables . . . . . . . . . . . . . 98
Table 27. Sample Organization Virtual Datacenter Trending Information. . . . . . . . . . . . . 99
VMware vCloud
1. What is a VMware vCloud?

1.1 Document Purpose and Assumptions
Architecting a vCloud is intended to serve as a reference for cloud architects. The target audience is VMware
Certified Professionals (VCP) familiar with VMware products, particularly VMware vSphere (vCenter Server,
ESXi, vShield Manager), VMware vCenter Chargeback, and VMware vCloud Director.
Before proceeding with the rest of this document, you should have read the Service Definition for the type of
cloud you are building, private or public. This document is not intended to be a substitute for detailed product
documentation, nor is it a step-by-step guide for installing a vCloud. You should have access to the following
documentation referred to throughout this document for step-by-step instructions on installing and configuring
various components.
To p i c
Referenced Document
Cloud Requirements
Requirements for a Cloud
vCloud Service Definitions
Service Definition for Public Cloud

Service Definition for Private Cloud
vCloud Implementations
Service Provider Public vCloud Implementation Example

Private vCloud Implementation Example
vCloud Director
vCloud Director Installation and Configuration Guide

vCloud Director Administrators Guide
vCloud Director Security Hardening Guide
vCloud API
vCloud API Specification

vCloud API Programming Guide
vSphere
vSphere Datacenter Administration Guide

vSphere Resource Management Guide
vShield
vShield Administration Guide
vCenter Chargeback
vCenter Chargeback Users Guide

Using vCenter Chargeback with VMware Cloud Director Technical Note
vCenter Orchestrator (vCO)
vCenter Orchestrator Developers Guide

VMware vCenter Orchestrator Administration Guide
vCenter Server 4.1 Plug-In API Reference for vCenter Orchestrator
vCloud Request Manager
vCloud Request Manager Installation and Configuration Guide

vCloud Request Manager Users Guide
Table 1. Reference Documentation
VMware vCloud
For further information, refer to the set of documentation for the appropriate product. For additional guidance
and best practices, refer to the Knowledge Base on vmware.com.
1.2 Cloud Computing and vCloud Introduction

A vCloud is VMwares cloud solution built on VMware technologies and solutions to deliver cloud computing.
Cloud computing is a new approach to computing that leverages the efficient pooling of on-demand, selfmanaged virtual infrastructure to provide resources consumable as a service.
Cloud computing can be delivered as three layers of service delivery:
Infrastructure as a Service (IaaS)
Platform as a Service (PaaS)
Software as a Service (SaaS)
This iteration of a vCloud focuses strictly on the IaaS layer.
The vCloud will build upon VMware vSphere by extending the robust virtual infrastructure capabilities to
facilitate delivery of infrastructure service via cloud computing.
1.3 vCloud Components

The VMware vCloud is comprised of the following components:

vCenter Chargeback
vCloud API
vCloud
Director
vShield Edge
vCloud
Connector
vCenter
Orchestrator
VMware vSphere
Figure 1. vCloud Overview
VMware vCloud
v C lo u d C o m p o n e n t
D e s cr i p t i o n
VMware vCloud Director (vCD)
Cloud Coordinator and UI. Abstracts vSphere resources.
vCloud API
Includes:
vCloud Director Server(s) (also known as cell)
vCloud Director Database
vCloud API, used to manage cloud objects
vCloud API
API used to programmatically interact with a vCloud
VMware vSphere
Underlying foundation of virtualized resources.

The vSphere family of products includes:
vCenter Server and vCenter Server Database
ESXi hosts, clustered by vCenter Server
Management Assistant
VMware vShield
Provides network security services

Includes:
vShield Manager (VSM) virtual appliance
vShield Edge* virtual appliances, automatically deployed by vCloud
Director
*The fully licensed version of vShield Edge includes optional features such as VPN and load
balancing that are not integrated with vCloud Director.
VMware vCenter Chargeback
Optional component that provides resource metering and reporting to

facilitate resource showback/chargeback
Includes:
vCenter Chargeback Server
Chargeback Data Collector
vCloud Data Collector
VSM Data Collector
VMware vCenter Orchestrator
Optional component that facilitates orchestration at the vCloud API

and vSphere levels.
VMware vCloud Request Manager
Optional component that provides provisioning request and approval

workflows, software license tracking, and policy-based cloud
partitioning.
VMware vCloud Connector
Optional component to facilitate transfer of a powered-off vApp in

OVF format from a local vCloud or vSphere to a remote vCloud
Table 2. vCloud Components
Other VMware or third-party products or solutions are not addressed in this iteration of a vCloud.
T ECHNICAL W HI T E P A P E R / 1 0
VMware vCloud
From an architectural view, the following diagram shows how the core vCloud components interrelate.
vCloud API
vCloud Director Cell
vCloud Director End Users

Web Console
vCloud Director Database
NFS Server
VMware vSphere
vCenter
Service
vCenter Chargeback
LDAP
vCenter Database
vCenter
Chargeback
Server
vSphere Client
vShield
VM
VM
VM
VM
VM
VM
VM
wa
re
ESX/ESXi
Hosts
VM
wa
re
VM
VM
VM
VM
VM
VM
vCloud
Agent
VM
VM
VM
Data
Collectors
vShield Manager and vShield Edge Virtual Appliances

VM
VM
VM
VM
wa
re
VM
wa
re
vCloud
Agent
VM
VM
VM
VM
VM
VM
vCloud
Agent
VM
VM
VM
VM
VM
VM
VM
wa
re
vCloud
Agent
VM
VM
VM
wa
re
vCloud
Agent
VM
VM
VM
vCenter
Chargeback
Database
VM
vCenter
Chargeback
Web Interface
vCloud
Agent
Datastores
Figure 2. Core vCloud Logical Architecture
1.4 vCloud Infrastructure

From an infrastructure perspective, a vCloud is built on a foundation of virtual infrastructure, whose components
are split between a management cluster and cloud consumer resources.
Cloud Consumer Resources
Management
Cluster
Compute
Storage
Networking
Virtual Infrastrucure
Figure 3. vCloud Infrastructure
In building a vCloud, we assume that all management components, such as vCenter Server and vCenter
Chargeback Server, will run in virtual machines.
As a best practice of separating resources allocated for management functions from pure user-requested
workloads, the underlying vSphere clusters will be split into two logical groups:
A single management cluster running all core components and services needed to run the cloud.
Remaining available vCenter clusters are aggregated into a pool called cloud consumer resources. These
clusters will be under the control of VMware vCloud Director. Multiple clusters can be managed by the same
vCenter Server or different vCenter Servers, but vCloud Director will be managing the clusters through the
vCenter Servers.
VMware vCloud
Reasons for organizing and separating vSphere resources include:

Ensuring that management components are separate from the resources they are managing.
Minimizing overhead for cloud consumer resources. Resources allocated for cloud use have little overhead
reserved. For example, cloud resource groups would not host vCenter virtual machines.
Dedicating resources for the cloud. Resources can be consistently and transparently managed and carved up,
and scaled horizontally.
Management
Cluster
Compute
Storage
Networking
The underlying vSphere Infrastructure will follow vSphere best practices. Design considerations specific to a
vCloud will be addressed accordingly in this document, organized by the vCloud management cluster and cloud
consumer resources.
1.4.1 vCloud Management Cluster
Management
Cluster
Compute
Storage
Networking
The management cluster will follow vSphere best practices to facilitate load balancing, redundancy, and high
availability.
1.4.2 Compute Resources
Compute resources for the management cluster will follow vSphere best practices where possible, including but
not limited to VMware DRS, HA, and FT.
To facilitate VMware HA, a cluster of three VMware ESXi hosts will be used. While additional hosts can be added,
three hosts supporting just vCloud management components should be sufficient for typical vCloud
environments. Detailed sizing guidance of the management cluster is provided later in this document. Use a
VMware HA percentage-based admission control policy in an N+1 fashion instead of dedicating a single host
for host failures or defining the amount of host failures a cluster can tolerate. This will allow the management
workloads to run evenly across the hosts in the cluster without the need to dedicate a host strictly for host
failure situations. Additional hosts can be added to the management cluster for N+2 or more redundancy but
this is not required by the current vCloud Service Definitions.
Use VMware HA (including VM Monitoring) and/or FT, where possible, to protect the management virtual
machines. vCenter Site Recovery Manager (SRM) can be used to protect components of the management
cluster. At this time, vCenter Site Recovery Manager will not be used to protect vCloud Director cells because
a secondary (DR) site is out of scope of the vCloud, and changes to IP addresses and schemas in recovered
vCloud Director cells can result in problems.
VMware vCloud
Unlike a traditional vSphere environment where vCenter Server is used by administrators to provision virtual
machines, vCenter Server plays an integral role in end-user self-service provisioning by handling all virtual
machine deployment requests by vCloud Director. Therefore, VMware recommends that vCenter Servers are
made available with a solution such as vCenter Heartbeat. Since FT is not supported for a multi-vCPU virtual
machine, this is another reason for using vCenter Heartbeat for high resiliency.
1.4.3 Storage Resources
Shared storage in the management cluster will be configured to include, but not limited to, the following:
Storage paths will be redundant at the host (connector), switch, and storage array levels.
All hosts in a cluster will have access to the same datastores.
1.4.4 Networking Resources
Host networking in the management cluster will be configured to include (but not limited to) the following:
Logical separation of network traffic for security and load considerations by type (management, virtual
machine, vMotion/FT, IP storage.
Network component and path redundancy.
At least 10GigE or GigE network speeds, if available.
Use of vNetwork distributed switches where possible for network management simplification. The architecture
calls for the use of vNetwork distributed switches in the user workload resource group, so it is a best practice
to use the vNetwork Distributed Switch across all of your clusters, including the management cluster.
Increasing the MTU size of the physical switches as well as the vNetwork distributed switches to at least 1524
(default is 1500) to accommodate the additional MAC header information used by vCloud Director Network
Isolation links. vCloud Director Network Isolation is called for by the Service Definition and the architecture
found later in this document. This needs to be done on the transport network for vCloud Director Network
Isolation. Failure to increase the MTU size could affect performance due to packet fragmentation affecting
network throughput of virtual machines hosted on the vCloud infrastructure.
1.4.5 Component Placement
Management components running as virtual machines in the management cluster include the following:
vCenter Server(s) and vCenter Database
vCloud Director Cell(s) and vCloud Director Database
vCenter Chargeback Server(s)
vShield Manager (one per vCenter Server)
Note: vShield Edge appliances are deployed automatically by vCloud Director through vShield Manager as
needed and will reside in the vCloud consumer resource clusters, not in the management cluster. They will be
placed in a system resource pool by vCloud Director and vCenter. For additional information on the vShield
Edge appliance and its functions, refer to the vShield Manager Administrator guides.
Optional management functions, deployed as virtual machines include:
vCenter Update Manager
vCenter Capacity IQ
VMware Management Assistant
vCenter Orchestrator (part of vCenter Server)
vCloud Request Manager and associated database
The optional management virtual machines are not required by the Service Definition but they are highly
recommended to increase the operational efficiency of the solution.
Database components, if running on the same platform, can be placed on the same database server. For
example, the databases used by vCloud Director, vCenter Chargeback, and vCloud Request Manager can be
consolidated on the same database server.
VMware vCloud
For more information on the resources needed by the virtual machines in the management cluster refer to the
Sizing section later in this document.
1.4.6 vCloud Consumer Resources
Management
Cluster
Compute
Storage
Networking
The cloud consumer resources represent vCenter clusters to host cloud workloads. These resources will be
carved up by vCloud Director. Well cover vCloud Director cloud constructs and definitions in the next section
first before drilling down on the compute, storage, and networking resources.
1.4.7 vCloud Logical Infrastructure
In summary, the vCloud logical architecture with vSphere resource separation is depicted as follows.
Management Cluster
VM
VM
VM
VM
VM
VM
VM
VM
vCloud Consumer Resources
wa
re
VM
VM
VM
VM
VM
VM
VM
wa
VM
VM
VM
VM
VM
VM
VM
VM
VM
wa
re
VM
VM
VM
VM
VM
VM
VM
wa
VM
VM
VM
VM
VM
VM
VM
VM
VM
wa
re
VM
VM
re
VM
re
VM
VM
VM
wa
VM
VM
VM
wa
vCloud Director Cells & vCloud Director

Database
VM
VM
VM
VM
VM
re
VM
VM
VM
VM
VM
wa
VM
VM
VM
VM
re
vCloud infrastructure virtual machine
VM
VM
VM
VM
wa
re
vCenter Servers & vCenter Database
VM
VM
VM
wa
re
re
vCenter Chargeback Servers

vShield Manager (1 per vCenter Server)
Optional Management Functions, deployed as virtual
machines
vCenter Update Manager
VM
VM
VM
VM
VM
VM
VM
VM
wa
re
VM
VM
VM
VM
VM
VM
VM
wa
re
VM
VM
VM
VM
VM
VM
VM
wa
re
VM
VM
VM
VM
VM
VM
wa
re
vCenter Capacity IQ
VMware Management Assistant
vCenter Orchestrator (part of vCenter
Server)
Space allocated to user workloads

vCloud infrastructure virtual machines (small footprint)
vShield Edge virtual appliances
No user workloads
Figure 4. vCloud Logical Architecture
The management cluster may also include virtual machines or have access to servers that provide infrastructure
services such as directory (LDAP/AD), timekeeping (NTP), networking (DNS, DHCP), and security (certificate).
Detailed considerations for sizing are addressed in the Sizing section.
VMware vCloud
The management cluster resides in a single physical site. vCloud consumer resources also reside within the same
physical site, ensuring a consistent level of service. Otherwise, latency issues might arise if workloads need to be
moved from one site to another, over a slower or less reliable network. For definition purposes, this cloud is
defined under the context of a single physical site, and does not span multiple sites.
Considerations for connecting clouds representing different sites are addressed later in this document.
Secondary DR sites are discussed later in the Disaster Recovery section of this document.
2. vCloud Director Constructs

VMware vCloud Director introduces logical constructs, such as virtual datacenters, and security boundaries,
such as organizations, to facilitate multi-tenancy consumption of resources.
The following diagram depicts the logical constructs within vCloud Director that abstract underlying vSphere
resources.
Organization A
Users
Catalogs
Organization B
Access Control
Users
Provisioning Policies
Catalogs
User Clouds
Organization vDCs
Access Control
Provisioning Policies
User Clouds
vApp
(VMs with vApp Network)
Organization vDCs
vApp
(VMs with vApp Network)
vSphere
vApp Network
Organization Network
External Networks
Organization vDCs
Provider vDC: Gold
Organization vDCs
Organization vDC
Provider vDC: Silver
Provider vDC: Bronze
Port Groups or
dvPort Groups
Resource Pools
Host Cluster
Datastores
Figure 5. vCloud Director Construct to vSphere Mapping
v C lo u d D i r e cto r C o n s tr u ct
Provider Virtual Datacenter
Logical grouping of vSphere compute resources

(attached vSphere cluster and one or more datastores)
for the purposes of providing cloud resources to
consumers.
Organization
A unit of administration that represents a logical

collection of users, groups, and computing resources,
and also serves as a security boundary from which
only users of a particular organization can deploy
workloads and have visibility into such workloads in
the cloud.
In the simplest term, an organization = an association
of related end consumers.
VMware vCloud
v C lo u d D i r e cto r C o n s tr u ct
Organization Virtual Datacenter
Subset allocation of a provider virtual datacenters

resources assigned to an organization, backed by a
vCenter resource pool automatically created by
vCloud Director. An organization virtual datacenter
allocates resources using one of three models:
Pay as you go
Reservation
Allocation
vApp Templates and Media Catalogs
A collection of available services for consumption.

Catalogs contain vApp templates (preconfigured
containers of one or more virtual machines) and/or
media (ISO images of operating systems).
External Network
A network that connects to the outside using an

existing vSphere network port group.
A network visible within an organization. It can be an

external organization network with connectivity to an
external network, and use a direct or routed
connection, or it can be an internal network visible
only to vApps within the organization.
vApp Network
A network visible within a vApp. It can be connected to

other vApp networks within an organization and use a
direct or routed connection, or it can be an internal
network visible only to virtual machines within the
vApp.
Network Pool (not shown in diagram)
A set of pre-allocated networks that vCloud Director

can draw upon as needed to create private networks
and NAT-routed networks.
Table 3. vCloud Director Constructs
3. vCloud Consumer Resources

3.1 Cloud Consumer Resources
Management
Cluster
Compute
Storage
Networking
The cloud consumer resources are dedicated vCenter clusters that host cloud workloads. These resources are
carved up by vCloud Director in the form of one or more provider virtual datacenters, which is a vCenter cluster,
and one or more attached datastores. Networking for the resource group will encompass vSphere networks
visible to the hosts in that cluster. Provider virtual datacenters are further carved up into organization virtual
datacenters, which are backed by vCenter resource pools.
VMware vCloud
Host Cluster
vCloud Consumer Resource
(Includes visible networking)
Datastore
Provider Virtual Data Center
Organization Virtual Data Center
Resource Pool
Figure 6. vCloud Consumer Resource Mapping
3.2 Establish Provider Virtual Datacenters

A provider virtual datacenter is a construct in vCloud Director that maps to a vSphere cluster or resource pool
and one or more datastores. When creating a provider virtual datacenter, take the following rules and guidelines
into consideration:
At least one provider virtual datacenter is required for a vCloud.
A provider virtual datacenter can map to one and only one cluster. Once a cluster is attached to a provider
virtual datacenter, it is no longer available for attachment to another provider virtual datacenter.
While it is possible to back a provider virtual datacenter with a resource pool instead of a cluster, the best
practice is to use a cluster. If additional hosts are later added to the cluster, the backed provider virtual
datacenter automatically grows as well. This is not the case if a resource pool is used. Also, since vCloud
Director manages vSphere resources by proxy through a vCenter Server and automatically creates resource
pools within vCenter as needed to instantiate organization virtual datacenters, using vCenter Server to create
resource pools or nested pools can go against the efficient allocation of resources by vCloud Director. Multiple
parent-level resource pools can also add unnecessary complexity and lead to unpredictable results or
inefficient use of resources, if the reservations are not set appropriately.
It is not possible to attach a second cluster to a provider virtual datacenter at this time. If additional compute
capacity is required, add more hosts in the vCenter cluster on the vSphere end.
One or more datastores can be attached to a provider virtual datacenter. A datastore can be assigned to
multiple provider virtual datacenters. As a best practice in segmenting storage, datastores should not be
shared by multiple provider virtual datacenters.
Create multiple provider virtual datacenters to differentiate computing levels or performance characteristics of
a service offering. Segment by capacity, availability, or performance type. An example of differentiating by
availability would be N+1 for a Bronze provider virtual datacenter vs. N+2 for a Silver provider virtual
datacenter.
As the level of expected consumption increases for a given provider virtual datacenter, add additional hosts to
the cluster from vCenter and attach more datastores.
As the number of hosts in the cluster backing a provider virtual datacenter approaches the halfway mark of
vSphere limits, consider implementing controls to preserve headroom. Do this well ahead of approaching the
cluster limits. For example, do not allow additional tenants for this particular virtual datacenter and utilize the
additional hosts to be added to address increased resource demand for the existing tenants.
If the cluster backing a provider virtual datacenter has reached the maximum number of hosts per vSphere
design guidelines, create a new provider virtual datacenter associated with a new cluster. A provider virtual
datacenter cannot span multiple host clusters.
VMware vCloud
Refer to the Service Definition for guidance on the size of vSphere clusters and datastores to attach when
creating a provider virtual datacenter.
Consider:
Expected number of virtual machines
Size of virtual machines (CPU, RAM, disk)
3.2.1 Public Cloud Considerations
Considerations for a public vCloud include creating multiple provider virtual datacenters based on tiers of
service that will be provided.
Since provider virtual datacenters only contain CPU, memory, and storage resources and those are common
across all of the requirements in the Service Definition for Public Cloud, you should create one large provider
virtual datacenter attached to a vSphere cluster that has sufficient capacity to run 1,500 virtual machines. You
should also leave overhead to grow the cluster with more resources up to the maximum of 32 hosts, should
organizations need to grow in the future.
If you determine that your hosts do not have sufficient capacity to run the maximum number of virtual machines
called out by the Service Definition for Public Cloud, then you will need additional provider virtual datacenters.
3.2.2 Private Cloud Considerations
Given that a provider virtual datacenter represents a vSphere cluster, it is commonly accepted that a single
provider virtual datacenter be established. Since provider virtual datacenters only contain CPU, memory, and
storage resources and those are common across all of the requirements in the Service Definition for Private
Cloud, you should create one large provider virtual datacenter attached to a cluster that has sufficient capacity
to run 400 virtual machines.
Refer to the Service Definition for Private Cloud for details on the service tier(s) called for. Should it be
determined that existing host capacity cant meet the requirement, or theres a desire to segment capacity along
the lines of equipment type (for example, CPU types in different provider virtual datacenters), then establish a
provider virtual datacenter for Pay-As-You-Go use cases and a separate provider virtual datacenter for the
resource-reserved use cases.
3.2.3 Provider Virtual Datacenter Special Use Cases
There are instances where a provider virtual datacenter must be viewed as special purpose in one way or
another. Special use-case provider virtual datacenters are a great example of what makes cloud computing so
flexible and powerful. The primary driver behind this need is to satisfy the license restrictions imposed by a
specific software vendor that stipulates that all the processors that could run specific software must be licensed
for it, regardless of whether or not they actually are running that software.
In order to meet the EULA requirements of such a software vendor, you can establish a purpose-specific
provider virtual datacenter, populated with enough sockets of processing power to meet the need but limited to
no more than what is needed in order to keep licensing costs down.
An example of this, in practice, is establishing an Oracle-only provider virtual datacenter. Since a provider virtual
datacenter is backed by a cluster, you can name a cluster for a special purpose and publish it to any and all
clouds that might need the service. You then maintain enough paid licenses to cover all the sockets in that
cluster, and you are covered under the EULA, because the guests can only run on one of the sockets in that
cluster/resource pool.
There is some level of user education needed to verify that all Oracle instances are deployed to the special
purpose virtual datacenter; that is, vCloud Director does not provide a way to prevent someone from incorrectly
deploying virtual machines. So, the enforcement has to be manual, typically through organizational processes.
In the following example, you could name the virtual datacenter with a descriptive name to deploy all Oracle
instances, or insert instructions in the vApp name so that users deploy to the correct virtual datacenter:
Oracle Database Use Only! PvDC
VMware vCloud
3.2.4 Compute Resources Considerations

Management
Cluster
Compute
Storage
Networking
All hosts in will be configured per vSphere best practices, similar to the management cluster. VMware HA will
also be used to protect against host and virtual machine failures.
Provider vDCs can be of different compute capacity sizes (number of hosts, number of cores, performance of
hosts) to support differentiation of compute resources by capacity or performance for service level tiering
purposes.
Organization vDCs in turn should be created based on what services are planned.
For a detailed look at how to size the vCloud, refer to the Sizing section later in this document.
The following table lists out the requirements for each of the components that will run in the vCloud Director
management cluster. For the number of virtual machines and organizations listed in the Service Definitions you
will not need to worry about scaling too far beyond the provided numbers.
It e m
v C PU
Memory
Sto rag e
N e tw o r k i n g
vCenter Server
8 GB
20 GB
100 MB
Oracle Database
16 GB
100 GB
1 GigE
vCloud Director x 2
(stats for each)
4 GB
10 GB
1 GigE
vCenter Chargeback
8 GB
30 GB
1 GigE
vShield Manager
4 GB
512 MB
100 MB
TOTAL
11
40 GB
161 GB*
3 GigE*
* Numbers rounded up or down will not impact overall sizing

Table 4. Component Requirements for a Management Cluster
For the table above, the Oracle Database will be shared between the vCenter Server, the vCloud Director cells,
and the vCenter Chargeback Server. Different users and instances should be used for each database instance
in-line with VMware best practices.
In addition to the storage requirements above, a NFS volume is required to be mounted and shared by each
vCloud Director cell to facilitate uploading of vApps from cloud consumers. The size for this volume will vary
depending on how many concurrent uploads are in progress. Once an upload completes the vApp is moved to
permanent storage on the datastores backing the catalogs for each organization and the data no longer resides
on the NFS volume. The recommended starting size for the NFS transfer volume is 250 GB. You should monitor
this volume and increase the size should you experience more concurrent or larger uploads in your environment.
VMware vCloud
3.2.5 Storage Resources Considerations

Management
Cluster
Compute
Storage
Networking
Shared storage in the consumer resources will be configured per vSphere best practices, similar to the
management cluster. Storage types supported by vSphere will be used. The use of RDMs in the vCloud
infrastructure is currently not supported and should be avoided.
Creation of datastores will need to take into consideration Service Definition requirements and workload use
cases, which will affect the number and size of datastores to be created. vCloud Director will assign datastores
for use through provider virtual datacenters, and only existing vSphere datastores can be assigned.
Datastores attached to provider virtual datacenters will be used for vCloud workloads, known as vApps. vSphere
best practices apply for datastore sizing in terms of number and size. Vary datastore size or shared storage
characteristic if providing differentiated or tiered levels of service. Sizing considerations include:
Datastore storage expectations:
Size a datastore sufficiently to allow for placement of multiple vApps, avoiding creating small datastores that
can house only one or two vApps. A few large datastores are preferred over many small datastores,
especially since consumers are not allowed to choose which datastores to place their workload on when
selecting a virtual datacenter with more than one datastore; vCloud Director will choose the datastore with
the most free space available.
What is the average vApp size x number of vApps x spare capacity? For example: Average virtual machine
size * # virtual machines * (1+ % headroom)
What is the average virtual machine disk size?
How many virtual machines are in a vApp?
How many virtual machines are to be expected?
How much spare capacity do you want to allocate for room for growth (express in a percentage)?
Will expected workloads be transient or static?
Datastore performance characteristics:
Will expected workloads be disk intensive?
What are the performance characteristics of the associated cluster?
Refer to the requirements in the Service Definition and size your datastores accordingly.
Additionally, an NFS share must be set up and made visible to all hosts for use by vCloud Director for
transferring files in a vCloud Director multi-cell environment. NFS is the required protocol for the transfer
volume. Refer to the vCloud Director Installation and Configuration Guide for more information on where to
mount this volume.
See the Workload Availability section for additional storage and storage I/O factors to take into account.
VMware vCloud
3.2.6 Networking Resources Considerations

Management
Cluster
Compute
Storage
Networking
Host networking for hosts within a provider vDC will be configured per vSphere best practices in the same
manner as the vCloud management cluster. In addition, the number of vNetwork Distributed Switch ports per
host should be increased from the default value of 128 to the maximum of 4096. Increasing the ports will allow
for vCloud Director to dynamically create port groups as necessary for the private organization networks created
later in this document. Refer to the vSphere Administrator Guide for more information on increasing this value.
Networking at the provider and organization virtual datacenter level is detailed in the next section on providing
cloud resources.
3.3 Multi-Site/Multi-Geo Clouds

vCloud Director is neither designed nor supported for multi-site deployments in the currently shipping version of
the product, due to potential issues with network latency and reliability. In an environment with multiple sites,
each site should be a separate cloud that can be potentially interconnected, rather than having a single cloud
that spans the sites.
Multi-site can mean a lot of different things to different audiences. Some providers would like to have one user
interface that encompasses all of their sites. Some providers dont mind having multiple interfaces but they
would like the same services available in each location. These scenarios are discussed here.
3.3.1 Scenario #1Common User Interface
In our example for scenario 1 we have two physical locations where we are providing cloud service and we want
both of them to be in the same vCloud Director interface. The user interface is provided from the vCloud
Director cells, which can sit in either location or in both locations. One of the vCloud Director cells will serve as
the proxy for a vCenter Server in one of the sites as illustrated below.
Site 1
vCD
vCD
vCD
Site 2
vCloud Director
vCenter Server
vCD
VM
VM
VM
VM
VM
VM
vCD
vCenter Server
VM
VM
vCD
wa
re
VM
VM
VM
VM
VM
VM
wa
re
Figure 7. Two Sites with Local vCloud Director Instances Managing Local vCenters
VMware vCloud
The local vCenter servers will control resources local to each site. This is a very logical setup of the infrastructure
until you look at some of the user flows.
If we are a user that is coming into site #1 requesting remote console access to a virtual machine in site #1 we are
not guaranteed to have all traffic stay in site #1. This is because we cannot control which vCloud Director cell is
the proxy for which vCenter Server. We could come into a vCloud Director cell in site #1 which would then have
to talk to the proxy for vCenter server #1 in site #2. That vCloud Director cell would then talk back to the vCenter
server in site #1 that would then finish setting up the remote console connection to the local ESXi host with the
virtual machine in question in site #1. Traffic at that time would then flow through the vCloud Director cell that
initiated the request in site #1. This is all illustrated below.
Site 1
vCD
vCD
vCD
vCenter Server
Site 2
vCD
VM
VM
VM
VM
VM
VM
vCD
vCenter Server
VM
VM
vCD
wa
re
VM
VM
VM
VM
VM
VM
wa
re
Figure 8. Remote Console Flow
One of the problems with this setup is how do you control which vCloud Director cell a user gets terminated on,
based on virtual machine and site specific data? Its next to impossible to continue to figure this out and provide
that logic to a load balancer. Another problem with the scenario is we need to have a central Oracle database for
all of the vCloud Director cells from both sites. This creates even more traffic on the link between the 2 sites since
the message bus in vCloud Director uses the Oracle database for communication. Overall this solution is less
than optimal and only suggested for cross-campus multi-site configurations where site-to-site communication
will not overwhelm the network and where network availability is highly reliable.
3.3.2 Scenario #2Common Set of Services
A more pragmatic approach to multi-site setups is to have a single vCloud Director setup in each of the sites that
is isolated from other sites. This solves the network cross-talk issue but it introduces even more problems of its
own. For example, how do you provide a common set of services across the different sites? How do you keep
organization names and rights as well as catalogs, networks, storage, and other information common across the
different sites? Currently there is no mechanism to do this in the currently shipping vCloud Director product.
Using other VMware technologies included in the vSphere suite of products you can synchronize cloud
deployments using automation scripts and provide common sets of services across locations.
In an enterprise, a private vCloud maps to a single site. Multiple vClouds can be connected using vCloud
Connector for offline vApp migrations. A public vCloud can be connected to form a hybrid cloud.
3.3.3 Suggested Deployment
Multi-site deployments are not officially supported by VMware at this time. However, if you are still going to
create a multi-site deployment, then the recommended way to deploy multi-site solutions is to set up an isolated
vCloud Director instance in each location. This isolated vCloud Director instance would include local vCloud
VMware vCloud
Director cells, vCenter Servers, an Oracle database instance, a vCenter Chargeback instance, and local vSphere
resources as illustrated in the picture below.
Site 1
vCD
vCD
vCD
vCenter Server
Site 2
vCD
vCD
vCenter Server
VM
VM
VM
VM
VM
VM
VM
VM
vCD
wa
re
VM
VM
VM
VM
VM
VM
wa
re
Figure 9. Two Sites with Isolated vCloud Director Instances
In order to keep the sites synchronized with organization and resource information, VMware encourages you to
create a set of onboarding scripts and workflows. These workflows would be used anytime you need to create a
new organization or a new resource for an organization and would drive that creation across all cloud sites. The
VMware cloud services organization can assist you in the creation of these customer specific workflows based
on templates the cloud practice already has. VMware cloud services has created these template workflows using
the vCenter Orchestrator product that is included with vSphere. By using the workflows for administrative
resource creation you can keep multiple clouds synchronized with organization resources.
3.3.4 Other Multi-Site Considerations
When creating multi-site configurations, there are resources out of the control of the vCloud setup that require
some thought. How do you set up networking between the sites? How do you handle IP addressing? It is for
these physical resource decisions, varying between customers, that we have not provided specific guidance on
in the reference architecture. Setting up these physical resources is also not included in the sample scripts
previously mentioned.
3.3.5 Merging Chargeback Reports
In our reference multi-site setup we included multiple vCenter Chargeback instances. In order to provide one
common bill or usage report to your cloud consumer you must aggregate all of the chargeback reports into one
report. You can leverage the vCenter Chargeback API as well as vCenter Orchestrator to pull chargeback reports
from each vCenter Chargeback server and consolidate them into one master report.
3.3.6 Synchronizing Catalogs
Synchronizing catalogs between sites is the most time consuming task. When setting up multiple cloud sites you
should designate one site as the master site for template creation and have all other sites be replication peers.
It is advisable to leverage native storage array replication to replicate the storage for the templates in each
catalog. Array replication can provide several benefits for long distance data movement including data
de-duplication and compression. Once the data is synchronized you can leverage the catalog synchronization
workflows provided by VMwarevCloud API to import the replicated templates into the appropriate catalogs in
VMware vCloud Director.
Synchronizing templates added at remote sites is out of scope for this version of the reference architecture.
This feature can be added selectively by engaging VMware Professional Services.
VMware vCloud
4. Providing Cloud Resources

4.1 Establish Organizations
A vCloud contains one or more organizations. Each organization represents a collection of end consumers,
groups, and computing resources. Users authenticate at the organization level, using credentials established by
an organization administrator locally within vCloud Director or LDAP. LDAP integration can be done at the cloud
system level or per organization. Before you can configure LDAP on a per organization level, you must configure
LDAP at the cloud system level. Set this up based on the cloud organizations requirements. Please see the
vCloud Installation and Configuration Guide on how to set up the LDAP service with vCloud Director.
Users in an organization consume resources by selecting vApps from a predefined catalog.
When creating organizations, the name of the organization will be used in the URL to access the GUI for that
organization. As an example, ACME would be accessed at https://<hostname>/cloud/org/ACME. You should
take care to avoid special characters or spaces in the organization name since that will affect the URL in
undesirable ways.
You can use the system defaults for most of the other organization settings. The one exception is leases, quotas,
and limits. There are no specific requirements called out by the Service Definition for leases, quotas, and limits.
The provider should set these values to whatever works best in their cloud.
4.1.1 Administrative Organization
A vCloud requires at least one organization. As a best practice, the first organization to be created should be an
administrative organization. This organization will own a master catalog of vApp templates that are published
and shared with all other (standard) organizations.
Administrators assigned to the administrative organization will also be responsible for creating official template
virtual machines for placement in the master catalog for other organizations to use. Virtual machines in
development should be stored in a separate development catalog that is not shared with other organizations.
As a note of reference, there is already a default System organization in the vCloud Director environment. The
administrative organization being created here is different from the built-in System organization since it can
actually create vApps and catalogs and share them.
Make sure that when you create the administrative organization you set it up to allow publishing of catalogs.
4.1.2 Standard Organizations
Create an organization for each tenant of the vCloud as necessary. Each of the standard organizations should be
created with the following considerations:
Cannot publish global catalogs
Use system defaults for SMTP
Use system defaults for notification settings
Use leases, quotas, and limits meeting the providers requirements
4.2 Establish Networking Options

Workloads in the cloud consumer resources will have network connectivity at two levels:
External networks, used to connect to the outside. These are mapped to vSphere networks.
Internal or NAT-routed networks, used to facilitate VM-to-VM communication within a cloud. These are backed
by vCloud Director network pools.
VMware vCloud
4.2.1 External Networks

An external network provides connectivity outside an organization through an existing, preconfigured
vSphere network port group. The vSphere port groups can be created using standard vSwitch port groups,
vNetwork Distributed Switch port groups, or the Cisco Nexus 1000V.
In a public vCloud, these preconfigured port groups will provide access through the Internet to customer
networks, typically using VPN or MPLS terminations.
When creating an external network, make sure to have sufficient vSphere port groups created and made
available for virtual machine access in the vCloud.
4.2.2 Network Pools
vCloud Director creates a private network as needed from a pool of networks to facilitate VM-to-VM
communication and NAT-routed networks. vCloud Director supports one of three methods to back network
pools:
vSphere port group. vCloud Director uses one of many existing, preconfigured vSphere networks. The
networks themselves can have VLAN tagging for additional security.
VLAN. vCloud Director automatically uses VLAN tagging from a range provided to segment networks to
create internal networks (organization and vApp networks) as needed. This assumes that vCloud Director and
all the managed hosts have access to the VLANs on the physical network.
vCloud Director Network Isolation. vCloud Director automatically creates internal networks using MAC-in-MAC
encapsulation.
The following table compares the three options for a network pool.
C o n s i d e rat i o n
v S p h e r e P o rt
Gr o u p B ac k e d
V L A N B ac k e d
v C lo u d
N e tw o r k
I s o l at i o n
B ac k e d
How it works
Isolated port groups

must be created and
exist on all hosts in
cluster
Uses range of available,

unused VLANs
dedicated for vCloud
Overlay network (with

network ID) created for
each isolated network
Advantages
Only option compatible

with Cisco Nexus
1000V
Best network
performance
Optionally requires one

VLAN per vCloud
Director Network
Isolation backed
network pool
vCloud Director creates

portgroups as needed
More secure than

VLAN backed option
vCloud Director creates
portgroups as needed
Disadvantages
Requires manual
creation and
management of
portgroups
Chance of running out

of VLAN IDs
Overhead required for

MAC-in-MAC
encapsulation
Possible to use a
portgroup that is in
fact not isolated
Table 5. Network Pool Options
VMware vCloud
Considerations when using a vSphere port group-backed network pool include:

Standard or distributed virtual switches may be used.
vCloud Director does not automatically create port groups. You must manually create these ahead of time for
vCloud Director to use.
Considerations when using a VLAN-backed network pool include:
Organization and vApp networks created by vCloud Director out of VLAN backed network pools are private to
an organization or vApp, respectively.
Hosts in the cluster backing the provider vDC used by the organization vDC must be connected to VLAN trunk
ports.
vNetwork distributed switches are required for all hosts and the cluster backing the provider vDC used by the
organization vDC that draws from the network pool.
vCloud Director creates port groups automatically as needed.
Considerations when using a vCloud Network Isolation-backed network pool include:
vNetwork distributed switches are required for all hosts and the cluster backing the provider vDC used by the
organization vDC that draws from the network pool.
Increase the MTU size of the physical switches as well as the vNetwork distributed switches to at least 1524 to
accommodate the additional MAC header information used by vCloud Director Network Isolation links. Failure
to increase the MTU size could affect performance due to packet fragmentation affecting network throughput
of virtual machines hosted on the vCloud infrastructure.
Specify a VLAN ID for the MAC-in-MAC transport network (this is optional but recommended for security).
Leaving this blank will default to VLAN 0.
vCloud Director creates port groups automatically on vNetwork distributed switches as needed.
Private networks backed by vCloud Director Network Isolation use fewer VLAN IDs.
Organization and vApp networks created by vCloud Director out of vCloud Director Network Isolation backed
network pools are private with respect to an organization or vApp, respectively.
4.2.3 Cisco Nexus 1000V Considerations
In vCloud Director 1.0, the Cisco Nexus 1000V is only supported with the vSphere port group-backed option for
network pools, which also happens to be the least flexible. This is also true for the VMware vNetwork Standard
Switch, while the vNetwork Distributed Switch will support all network types.
Cisco Nexus 1000V requires a vNetwork Distributed Switch and therefore vSphere Enterprise Plus licensing.
The Cisco Nexus 1000V is typically deployed in a vSphere environment to provide increased network visibility,
common management and advanced layer 2 security and quality of service functionality for virtual networking.
As vCloud Director ideally uses dynamic, automatically provisioned, isolated networks; internally this means
these requirements for management and security do not directly apply to network pools. However they are
relevant to the external networks where traffic enters and exits the vCloud.
The next sections walk go into detail on networking options for external networks and network pools, and
discuss public vs. private vCloud perspectives.
4.3 Establish Networking OptionsPublic vCloud Example

Referencing the Service Definition for a Public Cloud, all service tiers use a shared public Internet connection.
To fulfill this, create a single external provider network. Make sure to give the network a descriptive name, such
as Provider-Internet, for the case here. You will connect this external network to a vSphere port group that is
actually connected to the Internet. Make sure you have the IP information for the physical network you have
attached to, including the network mask, default gateway, and DNS information. Lastly, you will create a pool
VMware vCloud
of static IP addresses that will be consumed by vShield Edge appliances (that facilitate a routed connection)
each time you connect an organization network to this external network.
For sizing purposes, you should create a large enough IP address pool so that each of your organizations can
have access to an external network. Per the Service Definition, the estimated number of organizations for 1,500
virtual machines is 25 organizations, so make sure you have at least 25 IP addresses in your static IP pool. More
IP addresses should be set aside if you plan to allow inbound access into organizations.
4.3.2 Network Pools
In addition to access to external networks, each organization in a public vCloud will have organization-specific
private networks. vCloud Director instantiates Isolated L2 networks through the use of network pools.
Create a single large network pool for all organizations to share, and limit the use of this network pool when you
create each individual organization. The network pool created will use vCloud Network Isolation for separating
the traffic. This will use an existing vNetwork Distributed Switch previously created for connecting hosts. Use a
VLAN to further segregate all of the vCloud Director Network Isolation traffic in a transport network from the
rest of the infrastructure.
Because the network pools will be used by both the external organization network and private vApp networks,
you will need at least 11 networks in the network pool per organization. Ten of the networks in the pool will be for
the private vApp networks according to the Service Definition for a Public Cloud. One of the networks will be
used for the protected external organization network. Given the estimate of 25 organizations, you need at least
275 networks in the pool. There is a limitation of a maximum of 4096 networks in a network pool due to the port
limitation on the vNetwork Distributed Switch. Ephemeral ports in a vNetwork Distributed Switch are also
limited to 1016 per Switch and per vCenter server, further limiting the number of networks that can be
instantiated from a network pool. When connecting the network pool to a vNetwork Distributed Switch, make
sure you have enough free ports left on the switch (at least 275).
vCloud Datacenter
Organization ACME Corp.
Network Pool
Org Net:
ACME-Private
Private Internal
Org Net: ACME-Internet
Private Routed
Provider-Internet
Figure 10. Example Diagram of Provider Networking for a Public vCloud
4.3.3 Organization Networks

Create two different organization networks for each organization, one external organization network and one
private internal organization network. You can do this as one step in the vCloud Director UI wizard by selecting
the default (recommended) option when creating a new organization network. When naming an organization
network, it is a best practice to start with the organization name and a hyphen, for example, ACME-Internet.
VMware vCloud
Per the Service Definition for Public Cloud, the external network will be connected as a routed connection that
will leverage vShield Edge for firewalling and NAT to keep traffic separated from other organizations on the
same external provider network. Both the external organization network and the internal organization networks
will leverage the same vCloud Director Network Isolation network pool previously established. For both the
internal network and the external network, you will need to provide a range of IP addresses and associated
network information. Since both of the networks will be private networks, you can use RFC 1918 addresses for
both static IP address pools.
The Service Definition for Public Cloud defines a limit of external connections with a maximum of 8 IP addresses,
so you should provide a range of 8 IP addresses only when creating the static IP address pool for the external
network. For the private network, you can make the static IP address pool as large as desired. Typically, a full
RFC 1918 class C is used for the private network IP pool.
The last step is to add external public IP addresses to the vShield Edge configuration on the external
organization network. By selecting Configure Services on the external organization network, you can add 8
public IP addresses that can be used by that particular organization. These IP addresses should come from the
same subnet as the network that you assigned to the systems external network static IP pool.
Figure 11. Configure External IPs

It is important to note that vCloud Director is designed for secure multi-tenancy, and Layer 2 networks are not
shared between customers for organization and vApp networks. External networks can be dedicated to
customers (for example, MPLS VPNs) and vShield Edge is available to securely share networks such as a
common internet VLAN. These capabilities should be considered when determining whether the Cisco Nexus
1000V is required in a vCloud Director deployment.
Where it has been determined that the Cisco Nexus 1000V is a requirement, either operationally or technically,
the recommended approach is to use the Nexus 1000v for external networks and a VMware vNetwork
Distributed Switch for network pools. This approach has the advantage of providing advanced functionality
where required, without limiting the flexibility of vCloud Director networking.
VMware vCloud
Using the Cisco Nexus 1000V for vCloud Director network pools is not recommended because this requires
increased administrative overhead; the Cisco Nexus 1000V works only with vSphere port group-backed network
pools and introduces scalability limits. In this model, organization and vApp networks cannot be created
dynamically because port profiles need to be defined on the Cisco Nexus 1000V before being added to vCloud
Director. To maintain isolation within these internal networks, each port group will need to be configured with a
VLAN ID, which given the limit of 512 active VLANs across all virtual ethernet module (VEMs) managed by a
Cisco Nexus 1000V, also potentially limits the total number of networking pools. This is particularly true if one
virtual supervisor module (VSM) is managing VEMs in multiple vCloud resource groups, and the 802.1q standard
itself is limited to 4096 VLANs.
vCloud Director Network Isolation backed-network pools is an approach to address these scalability limits by
providing isolation of internal vCloud Director networks using MAC-in-MAC encapsulation over a transport
network. Instead of individual VLANs, vCloud Director Network Isolation network IDs are dynamically assigned
to these encapsulated networks by vCloud Director. VLAN backed network pools is another option where a
range of VLAN IDs is allocated to vCloud Director, which will then assign these VLANs to organization and vApp
networks, as required. So while it shares the same scalability limits as port group backed, it reduces manual
administrator setup. VLAN and vCloud Director Network Isolation backed network pools are only supported
with the vNetwork Distributed Switch.
The following diagram illustrates the recommended deployment model with the Cisco Nexus 1000V, used for
external networks only.
Resource Group
VMware vSwitch
Mgmt Port
Group
Service Port
Group
VLAN 10
pNIC
VLAN 20
pNIC
Core
Physical
Switching
Nexus 1000v vDS

Org A External
VLAN 110
Org B External
VLAN 120
Org C External
VLAN 130
Org A VPN
VLAN 140
Org B VPN
VLAN 150
pNIC
pNIC
Core
Physical
Switching
VMware vDS
OrgNet1
VLAN 210
OrgNet2
VLAN 220
OrgNet3
VLAN 230
VCD-NI Pool A
VLAN 298
VCD-NI Pool B
VLAN 299
pNIC
pNIC
Figure 12. vCloud Director Logical Networking w/ Cisco Nexus 1000V
VMware vCloud
4.4 Establish Networking OptionsPrivate vCloud Example

In general, for a private vCloud, the networking needs are comparable to that of a public vCloud and oftentimes
simplified, representing a subset. As such, direct connections from inside the organization to the networking
backbone provided by the enterprise are all that is necessary. This is analogous to extending a wire from the
network switch that contains the network or VLAN to be used all the way through the cloud layers to the
organization and into the vApp. One of these direct networks must be established for each network or VLAN to
be used in the private vCloud.
Enterprise vCloud
Organization Software Design
Network Pool
Org Net:
Internal Network
Private Internal (optional)
Org Net: External Access
Private Direct
Corporate Backbone
Figure 13. Example Diagram of Provider Networking for a Private vCloud
An important differentiation in a private vCloud vs. a public vCloud is the external network and organization
external network.
At least one external network is required to enable organization external networks to access resources outside
of the vCloud Director resourcesthe Internet for public cloud deployments and an internal (local) network for
private cloud deployments. It is a network that already exists within the address space used by the enterprise.
To establish this network, follow the wizard, filling in the network mask, default gateway and other specifications
of the LAN segment as required. When building this, specify enough address space for use as static
assignments, as this is where vCloud Director draws Public IP Pool addresses from. A good starting range is 30
addresses that do not conflict with existing addresses in use, or ranges already committed for DHCP.
Note: Static IP Pool address space is not used for DHCP, but the function is similar to that. This pool will be used
to provision NAT-type connectivity between the organizations and the cloud services below it.
4.4.2 Network Pools
You will need a network in the network pool for every private organization network and external organization
network in the vCloud environment. The Service Definition for a Private Cloud calls for one external organization
network and the ability for the organization to create private vApp networks. Since there is no minimum called
out in the Service Definition for the number of vApp networks a good number of networks to start out with is 10
per organization. Make your network pool as large as the number of organizations times 10.
VMware vCloud
4.4.3 Organization Networks

At least one organization external network is required to connect vApps created within the organization to other
vApps and/or the networking layers beyond the Private vCloud.
To accomplish this, create an external network in the Cloud Resources section (under Manage & Monitor of the
System Administration section of the vCloud Director UI). In the wizard, be sure to select a direct connection.
This external network maps to an existing vSphere network for virtual machine use as defined in the External
Networks section (above).
Other networking options are available, like a routed organization external network, and could be used, but add
complexity to the design that is normally not needed. For the purpose of this design there are no additional
network requirements. For more information on adding additional network options, refer to the vCloud Director
Administrators Guide.
The Cisco Nexus 1000V is applicable as a switching fabric in the Enterprise. The caveats expressed in the public
vCloud section for Cisco Nexus 1000V also apply to the private vCloud. The primary difference is that the
networking rails used as external networks will likely be simpler that those shown for a public vCloud. For
example, theres likely to be only a single backbone LAN to connect to thats also the path to the Internet, as
compared to a public vCloud where there will be multiple, distinct network paths.
In summary, the Cisco Nexus 1000V is applicable to private vCloud implementations as a switching backbone for
external networks only, to represent the one or more core LANs that lead to the rest of the business and/or
Internet.
4.5 Establish Organization Virtual Datacenters

An organization virtual datacenter allocates resources from a provider virtual datacenter and makes it available
for use for a given organization. Multiple organization virtual datacenters can take from the same provider virtual
datacenter. An organization can have multiple organization virtual datacenters.
Resources are taken from a provider virtual datacenter and allocated to an organization virtual datacenter using
one of three resource allocation models:
Pay as you go. Resources are only reserved and committed for vApps as vApps are created. There is no upfront
reservation of resources.
Allocation. A baseline amount (guarantee) of resources from the provider virtual datacenter is reserved for
the organization virtual datacenters exclusive use. An additional percentage of resources is available to
oversubscribe CPU and memory, but this taps into compute resources that are shared by other organization
virtual datacenters drawing from the provider virtual datacenter.
Reservation. All resources assigned to the organization virtual datacenter are reserved exclusively for the
organization virtual datacenters use.
With all of the above models, the organization can be limited to deploy a certain number of virtual machines.
Or, this can also be set to unlimited.
The first organization virtual datacenter to be created should be an administration organization virtual
datacenter for use by the administration organization. The allocation model is set to Pay as you go so as not to
take resources from other organization virtual datacenters until they are needed.
Subsequent organization virtual datacenters should be created to serve the organizations previously
established. In selecting the appropriate allocation model, the Service Definition and organizations use cases of
workloads should be taken into consideration. Take into account anticipated workload capacity, discussed
further in the Capacity Management section.
VMware vCloud
4.5.1 Public vCloud Considerations

The organization virtual datacenter allocation model maps directly to a corresponding vCenter Chargeback
billing model:
Pay as you go. Pricing can be set per virtual machine, and a corresponding speed of a vCPU equivalent can be
specified. Billing is unpredictable as it is tied directly to actual usage.
Allocation. Consumers are allocated a baseline set of resources but have the ability to burst by tapping into
additional resources as needed, but are typically charged at higher rates for exceeding baseline usage. This
model will result in more variable billing but allows for the possibility of more closely aligning variable
workloads to their cost.
Reservation. Consumers are allocated and billed for a fixed container of resources, regardless of usage. This
model allows for predictable billing and level of service, but consumers may pay for a premium if they do not
consume all their allocated resources.
These allocation models also map directly to the service tiers found in the Service Definition for a Public Cloud.
The Basic vDC model will use the Pay-as-you-go allocation model since instances are only charged for the
resources they consume and there is no commitment required from the consumer. The Committed vDC model
will use the Allocation Pool model since the consumer is required to commit to a certain level of usage but is also
allowed to exceed that usage. The Dedicated vDC model will use the Reservation Pool model since this service
tier requires dedicated and guaranteed resources for the consumer.
An option to enable thin provision allows provisioning virtual machines using thin disks to conserve disk usage.
vSphere best practices apply in the use of thin-provisioned virtual disks.
The Service Definition for Public Cloud provides detailed and descriptive guidance on how much a provider
should charge for each service tier. Chargeback functionality is provided by VMware vCenter Chargeback, which
is integrated with VMware vCloud Director. You can reference the VMware vCenter Chargeback Users Guide for
information on how to customize the individual reports generated.
For further information, refer to the vCloud Chargeback Models Implementation Guide, which details how to set
up vCloud Director and vCenter Chargeback to accommodate instance-based pricing (pay as you go),
reservation-based pricing, and allocation-based pricing.
4.5.2 Private vCloud Considerations
The organization virtual datacenter allocation model used depends on the type of workloads to be expected.
Pay as you go. A transient environment where workloads are repeatedly deployed and undeployed, such as a
demonstration or training environment, would be suited for this model.
Allocation. Elastic workloads that have a steady state but during certain periods of time surge due to special
processing needs would be suited for this model.
Reservation. Since a fixed set of resources are guaranteed, infrastructure-type workloads that demand a
predictable level of service would run well using this model.
When an organization virtual datacenter is created in vCloud Director, vCenter Server automatically creates child
resource pools with the appropriate resource reservations and limits, under the resource pool representing the
provider virtual datacenter.
As part of creating an organization virtual datacenter, a storage limit must be set unless you are using the Pay as
you go allocation model, which defaults to unlimited.
For the purpose of this architecture there will be no limit on storage consumed by the vApps since we are
providing static values for the individual virtual machine storage and we are also limiting the number of virtual
machines in an organization.
An option to enable thin provision allows provisioning virtual machines using thin disks to conserve disk usage.
vSphere best practices apply in the use of thin-provisioned virtual disks. This feature can save substantial
amounts of storage and have very little performance impact on workloads in the vCloud infrastructure. It is
recommended to enable this feature when creating each organization. For more information about this feature,
refer to the vCloud Director Administrators Guide or the VMware knowledge base.
VMware vCloud
vCloud Request Manager can be used to facilitate policy-based creation of organization virtual datacenters in a
private cloud to automate and standardize their creation through the creation and use of a organization virtual
datacenter template (or blueprint) to deploy new organization virtual datacenters.
4.6 Create vApp Templates and Media Catalogs

The way to consume services in a cloud environment is from a catalog. Catalogs are stored in an organization
virtual datacenter.
The administrative organization virtual datacenter will have two catalogs:
Internal. Used for developing and staging new vApps and media.
Master. Published and shared to all other organization virtual datacenters.
Organizations will use the master catalog that has been published from the administrative organization virtual
datacenter with the default cloud templates. In addition, organizations will have a private catalog created by the
organization administrator and used for uploading new vApps or media to the individual organization.
vApp templates are used to deploy actual vApps. Guest customization can be applied during deployment and
can facilitate the joining of vApps to domains. See the Deploy vApps section.
There are no other configuration requirements for the catalogs or templates in this cloud architecture. Refer to
the Service Definition for a full listing of recommended templates. Typically the templates can be blank virtual
machines for installing OSs or virtual machines with applications.
4.6.1 Auto-Joining Active Directory Domains
vCloud Director supports guest customization when deploying vApps. Guest customization can be configured
to automatically join an Active Directory domain for Windows guests.
4.6.2 Establish Policies
During the creation of an organization, you can set policies around the number of deployed and stored virtual
machines:
Deployed virtual machines refers to the number of running virtual machines.
Stored virtual machines refers to the total number of virtual machines including virtual machines that are not
powered on.
You can also specify runtime policies to control vApps and vApp templates in an organization virtual datacenter.
Specify the maximum length of time vApps and vApp templates can run and be stored in the organization
virtual datacenters:
The runtime lease can be set to allow vApps or vApp templates to run for a defined period of time after which
time vApps will be powered off, or set to never expire.
The storage lease can be specified, allowing vApps or vApp templates to be stored for a defined period of time,
after which time vApps or vApp templates will be automatically cleaned up, or set to never expire.
When any option for storage lease (with the exception of never expire) is selected, the storage will be
automatically cleaned up. Additional options include:
Permanently deleted. After the specified period of time, the vApps or vApp templates will automatically be deleted.
Moved to expired items. This flags the vApps or vApp templates for deletion, which hides them from users so
that they can no longer be used, allowing an administrator to remove them.
The Service Definition for the Public Cloud has specific requirements for the maximum number of virtual
machines each organization can have based on size. Refer to the Service Definition for the Public Cloud for the
maximum virtual machine count for each of the three tiers of reservation pools.
VMware vCloud
4.6.3 Accessing your vCloud

Each organization should have a public URL configured to access the organizations cloud portal using vCloud
Director. These URLs will have the format of https://<vCD-cell-hostname>/cloud/org/<org-Name>. Each time a
user of an organization logs in they should point their browser to the organization-specific URL.
4.6.4 Deploy vApps
vApps can now be deployed from a catalog of vApp templates.
4.6.5 Employ Chargeback or Showback
In a public vCloud, chargeback is essential in accurately metering consumer usage and recouping costs to ensure
profitability.
In a private vCloud, IT does not necessarily have the same cost pressures as a public vCloud service provider. IT
may also not have chargeback procedures or policies in place, as chargeback typically is a financial policy. An
alternative to chargeback is showback, which merely attempts to raise awareness of consumption usage and
cost without involving formal accounting procedures to bill the usage back to the consumers department.
To align consumer behavior with the actual cost of the resources being consumed, utilize the vCenter
Chargeback reports to provide resource and financial transparency. Without showback or chargeback,
consumers do not have the awareness of the actual cost of the resources they have consumed and thus have
little incentive to change their consumption patterns. Cloud computing resources can be easily spun up, and with
the exception of deployment policies dictating resource leases, there are no disincentives or penalties to curb
excessive use. Showback or chargeback will expose heavy or demanding users.
5. Extending vCloud Capabilities

5.1 Core vCloud Components
The core vCloud components include vSphere, vCloud Director, vShield Edge, and vCenter Chargeback.
vCenter Chargeback
vCloud API
vCloud
Director
vShield Edge
vCloud
Connector
vCenter
Orchestrator
VMware vSphere
Additional components to extend the vCloud capabilities are discussed further below.
5.2 vCloud Request Manager

vCenter Chargeback
vCloud API
vCloud
Director
vShield Edge
vCloud
Connector
vCenter
Orchestrator
VMware vSphere
VMware vCloud
vCloud Director provides a cloud portal whereby end consumers can self-provision their own workloads. For
environments where a formal provisioning process, including a request /approval mechanism, is needed, vCloud
Request Manager can be used as a front-end to vCloud Director. vCloud Request Manager 1.0 is largely intended
for private vClouds due to the fact that vCloud Request Manager requires cloud system privileges to vCloud
Director and therefore would not be ideal in a multi-tenant public cloud.
vCloud Request Managers primary capabilities include:
Provisioning with approvals
Software license tracking
Policy-based cloud partitioning (new organization creation)
In relation to the reference architecture, this new component sits in the functional stack above vCloud Director as a
new user entry point/UI, but not obscuring it entirely, as to allow for vCloud Director to be an entry point as well.
vCloud Request Manager is a front-end to the default vCloud Director portal to be exposed to those users who
need the policy-based approval/provisioning/tracking functionality over the more freeform vCloud Director UI.
vCloud Request Manager adds a vApp request/approval cycle to vCloud Director. This adds a flexible
mechanism by which vApps in catalogs can be requested, approved, and provisioned, based on business needs.
The included workflows can be modified as needed to match these requirements.
vCloud Request Manager adds simple software asset tracking and pairs these assets with vApps in the catalog.
The quantity of available licenses of a particular package that is being tracked is decremented each time a vApp
with that package is provisioned, and incremented when the vApp is destroyed. This license-to-vApp
relationship is managed manually inside vCloud Request Managers workflow engine.
Lastly, vCloud Request Manager provides easy creation tools for new private clouds with approval cycles
included. This enables the quick creation of new cloud instances (organizations) based on blueprints of cloud
policies and parameters.
One obvious application for this technology is adding lifecycle management to vApps provisioned in the private
clouds, a solution formerly covered by vCenter Lifecycle Manager.
vCloud Request Manager also provides a simplified interface for requesting resources from multiple cloud
installations. vCloud Request Manager can be explored as an option if you require a single interface for all of
your cloud installations. vCloud Request Manager does not provide the full richness of vApp management in a
cloud environment. Several third party solutions exist as well that can provide one interface to multiple cloud
installations.
Refer to the vCloud Request Manager Installation and Configuration Guide for specific installation requirements.
vCloud Request Manager runs in its own virtual machine separate from vCloud Director.
Note that several objects in vCloud Request Manager map to corresponding vCloud Director objects but do not
use consistent terms. They include:
organization in vCloud Request Manager refers to a vCloud Director system instance
location in vCloud Request Manager refers to an organization in vCloud Director
cloud in vCloud Request Manager refers to an organization vDC in vCloud Director
VMware vCloud
5.3 vCloud API

vCenter Chargeback
vCloud API
vCloud
Director
vShield Edge
vCloud
Connector
vCenter
Orchestrator
VMware vSphere
The vCloud API allows for interacting with a vCloud and can be used to facilitate communication with vCloud
Director using a UI other than the portal included with vCloud Director. For example, vCloud Request Manager
communicates with vCloud Director using the vCloud API. The vCloud API is the cornerstone to federation and
ecosystem support in a vCloud environment. All of the current federation tools talk to the vCloud environment
through the vCloud API. The ISV ecosystem also uses the vCloud API to allow their software to talk to vCloud
environments. It is very important that a vCloud environment exposes the vCloud API to the cloud consumer.
Currently, VMware vCloud Director is the only software package that exposes the vCloud API. In some
environments, vCloud Director is deployed behind a portal or in another location not readily accessible to the
cloud consumer. In this case there needs to be an API proxy or relay in to have the vCloud API exposed to the
end consumer.
Due to the value of the vCloud API, some environments may wish to meter API usage and charge extra for it to
customers. Protecting the vCloud API through audit trails as well as API inspection is also a good idea. Lastly,
there are several cases where cloud providers may wish to extend the vCloud API with new features.
In order to aid in several of the vCloud API use cases discussed, the cloud provider may wish to implement an
API proxy. The vCloud API is a REST-based service that contains XML payloads. For this reason any suitable XML
gateway can be used to proxy the vCloud API. There are several third party solutions on the market that excel in
XML gateway services today. VMware has begun to partner with some of these vendors to develop joint
guidance on how to deploy their solutions in a vCloud Director environment. For the latest information on these
efforts and collateral, please contact your local VMware vCloud specialist.
5.4 vCenter Orchestrator

vCenter Chargeback
vCloud API
vCloud
Director
vShield Edge
vCloud
Connector
vCenter
Orchestrator
VMware vSphere
vCenter Orchestrator (vCO) is a system for assembling operational workflows. The primary benefit of vCloud
Orchestrator is to coordinate multiple systems to achieve a composite operation that would have taken several
individual operations on different systems. It is not meant to replace the APIs that it in turn calls; rather, it is
meant to bring them together when necessary. In general, if an operation uses only one underlying system,
direct access to that system should be considered for efficiency and complexity reduction. In the vCloud use
case, orchestration can be used to automate highly repetitive tasks to avoid manual work and potential errors.
vCloud Orchestrator has a plug-in framework. There are plug-ins for VMware products such as vCenter Server,
vCloud Director, vCenter Chargeback. Thus, vCloud Orchestrator can orchestrate workflows at the VIM API, VIX
API, vCloud API, and Chargeback API levels.
VMware vCloud
There are three main categories of use cases that vCloud Orchestrator can help satisfy:
Cloud administration operations
Organization administration operations
Organization consumer operations
5.4.1 Cloud Administration Orchestration Examples
Here are some example use cases that highlight the value of vCloud Orchestrator to the cloud owner. These use
cases are primarily focused on infrastructure management and the control of the resource allocation process.
First, consider the case of a provider who wants to bring a new customer into their vCloud. The major steps
would be to create a new organization, users (possibly imported from Active Directory), networks, virtual
datacenters, and catalogs. The provider may also want to set up a recurring chargeback report so the tenant
can be billed, and possibly send an email notification to the tenant advising them that their new cloud
environment is ready.
Another example would be the case of a tenant request for additional external network capacity. In this case, the
provider may want to automate the creation of the network, which would include name generation, identification
and allocation of available VLAN and IP address range, configuration of the network switch and cloud perimeter
firewall, creation of the external network in vCenter, and finally allocation of the external network to the tenants
organization.
5.4.2 Organization Administration Orchestration Examples
There are operational tasks within the tenants organization that can benefit from automation as well. These are
typically tasks that address the vApp and virtual machine lifecycle management tasks including creation,
configuration, routine maintenance, and decommissioning.
Consider the case of virtual machine creation in an environment that uses Active Directory to identify services
such as authentication and printing. After deployment, the virtual machine must join the Active Directory
domain. In many cases it is preferable to use an organization unit (OU) other than the default Computers
Container. vCloud Orchestrator could be used to create the virtual machines computer account in the proper OU
prior to virtual machine deployment, insuring that the computer account name is unique and residing in the
proper OU. Similarly, when the virtual machine is decommissioned, the entry in the OU can be removed as part
of the same workflow.
Another example is the case where an organization administrator would like to manage recurring updates to a
software package or configuration element across several virtual machines in a single operation. In this case, a
workflow could be created to accept a list of systems and a source for the software or configuration as
parameters, and carry out the activity on each system.
5.4.3 Cloud Consumer Operation Orchestration Examples
These operations generally fall into the category of tasks that the organization administrator wants to offload as
a self-service operation. Performing the operation as a vCloud Orchestrator workflow provides an easy way to
expose the operation to a customer via the built-in web portal or via a customized portal that leverages the
web-services API. Many of the operations in this category can be satisfied directly via the vCloud Director UI;
however, there are candidates that affect multiple systems or fit better into a customer portal, which may be
better implemented as an orchestration workflow. None of this is exposed to the cloud consumer, which makes it
somewhat difficult. This has to be initiated by the cloud provider using the vCenter Orchestrator Client, unless
the provider creates a portal to front-end vCenter Orchestrator.
Some examples of these types of use cases would include resetting of system or user account passwords on
virtual machines using the VIX plug-in, putting a load balanced service into maintenance mode by stopping the
service, removing it from the load balancing pool and disabling monitors, loading certificates into virtual
machines, and deploying instances of custom applications from the organizations catalog.
vCloud Orchestrator can be used to create custom workflows at the vCloud API and VIM levels. vCloud Request
Manager is an alternative to vCloud Orchestrator that has built-in workflow functionality that integrates with
vCloud Director through the vCloud API.
VMware vCloud
For additional information on topics on vCO installation and configuration and workflow solution development,
refer to the vCenter Orchestrator v4.1 documentation set at http://www.vmware.com/support/pubs/
orchestrator_pubs.html.
5.5 vCloud Connector

vCenter Chargeback
vCloud
Director
vCloud API
vShield Edge
vCloud
Connector
vCenter
Orchestrator
VMware vSphere
As more clouds are stood up, several clouds from different sites within a private enterprise can form a larger
cloud, or a private and public cloud can form a hybrid cloud. Cloud consumers need a way to migrate workloads
in a federated cloud.
vCloud Connector (vCC) solves this problem by allowing you to perform migrations from all of your public and
private clouds and obtain a consistent view of them from a single interface. vCloud Connector needs to be
installed by cloud administrators but can be used by administrators and end users alike to view and manage
workloads. Once vCloud Connector has been deployed to a vSphere host and registered with a vCenter Server,
end users can access the vCloud Connector under Solutions and Applications from the vSphere Client where
the OVF was deployed.
5.5.1 vCloud Connector Placement
There are two considerations on where to place your vCloud Connector appliance.
The virtual appliance must be deployed to a vCenter Server that the target users can access via the vCenter
Client. The only user access is via the vSphere Client so users of vCloud Connector must have the right to login
to this vCenter Server.
Workload copy operations use the vCloud Connector appliance as a middleman, so network latency and
bandwidth between clouds needs to be considered. In some cases it may be preferred to run multiple
instances of vCloud Connector across multiple vCenter Servers to avoid network latency or consuming
excessive bandwidth.
Local vCloud or vSphere
vCloud
Connector
UI
Remote vCloud
vCloud Director
X
REST/HTTP(S)
REST/HTTP(S)
vCloud Connector
Appliance
vSphere
Client
VM
VM
VM
VM
vCloud API
REST/HTTP(S)
vCloud Director
Y
wa
re
vCenter Server
A
ESXi Host
VIM API
SOAP/HTTP(S)
vCenter Server
B
Figure 14. vCloud Connector Architecture
VMware vCloud
5.5.2 vCloud Connector Example Usage Scenarios

vCloud Connector can support a number of workload migration use cases. The following examples assume
migration of a vApp, comprised of one or more virtual machines:
Copying a vApp from vSphere to a vCloud
Copying a vApp from a private vCloud to a public vCloud
Copying a vApp from a vCenter to another vCenter.
Even in environments not running vCloud Director, vCloud Director can still be used to copy and move vApps.
As long as both vCenter Servers are added as clouds in vCloud Director, you can freely move workloads
between them.
5.5.3 vCloud Connector Limitations
The use of vCloud Connector to copy and migrate vApps is subject to the following limitations:
Currently there is no way to have predefined clouds appear in vCloud Connector. Each user must manually add
all clouds to vCloud Connector that they intend intend to access. There are no clouds defined by default so that
the user can add only the clouds they care to see.
Traffic to and from the vCloud Connector appliance is not WAN optimized so it is not ideal to migrate
workloads over WAN links even if sufficient bandwidth exists. To avoid this scenario it is preferred to have
vCloud Connector appliances installed in locations to avoid having to traverse WAN links as much as possible.
There is currently no way to limit which clouds can be added to a vCloud Connector instance so you must
instruct your users to only use the proper vCloud Connector instance for their needs.
All workloads being transferred are copied to a staging area on the vCloud Connector appliance before being
copied to the destination cloud. This area is 20GB by default, which means the largest virtual machine that can
be copied must be smaller in disk space than that. The ability to easily resize this staging area will be available
in an upcoming update.
vCloud Connector is designed to give you a consistent view of your workloads across multiple clouds and
migrate those workloads. Therefore, you will still need to use the vCenter Client and/or login to the vCloud
Director to manage your workloads.
vApps/virtual machines must be powered off for migration, that is, the workloads must be offline. Hot
migrations are currently not available. The vApp networking configuration will also need to be modified before
powering on the virtual machines.
VMware vCloud
6. Managing the vCloud

6.1 Monitoring
Monitoring the components of a vCloud Director implementation is essential to the health of a vCloud
environment and meeting customer expectations. This section provides recommendations on what systems and
associated objects to monitor, and what readily available tools to extract health-related metrics. Details of
specific limits or thresholds are not identified, as they can be found in the detailed product documentation. This
document will not provide specifics on setting up a monitoring solution, as service providers and enterprises
may have very different monitoring solutions in place to be integrated.
6.1.1 Management Cluster
Monitoring the management cluster components will follow the same best practices as monitoring vSphere. As
part of this, a centralized monitoring tool such as Hyperic HQ Enterprise can be used to monitor the core objects
(Oracle Server, SQL Server, Active Directory Server, DNS Server, Red Hat Enterprise Linux Server, Windows
Server) that are needed to run a vCloud environment. SNMP and SMASH are not supported for monitoring
vCloud Director cells, however, SNMP can be integrated from vCenter. Alternatively, cells can be monitored
through integration with a third party monitoring platform via JMX Beans. Note that JMX Beans monitoring is
only the start. vCloud and vSphere API provide a significant amount of details that can be used from a health
and capacity management.
6.1.2 Cloud Consumer Resources and Workloads
Monitoring the cloud consumer resources and workloads will also follow best practices for monitoring vSphere.
However, additional cloud-specific considerations are as follows.
vShield Edge
vShield Edge appliances are self-contained environments that are stateless in nature. There is a health check
API call you can make to a vShield Edge appliance to determine if it is functioning correctly. If the API returns
negative, then you should initiate a reboot of the vShield Edge device. At the time of reboot, configuration
information will be updated from the vShield Manager and the vShield Edge device will continue to function
properly.
Cloud Consumer Workloads
It may be desirable to monitor workloads provisioned by cloud consumers. vCloud Director does not provide any
built-in monitoring of workloads for availability or performance. Several third party solutions are available to
monitor vSphere resources and workloads running on vSphere; however, not all of these solutions may work all
of the time when vCloud Director is in use. Isolated networking in vApps may prevent monitoring tools from
acquiring performance or availability information of a vApp. Furthermore, vApps may be provisioned and
de-provisioned or power cycled at any time by a cloud consumer and these actions may create false positives
in the monitoring environment. Until there are solutions that are fully integrated with vCloud Director on the
market, it may be difficult to provide detailed monitoring for cloud consumer workloads.
For detailed guidance on how to monitor a vCloud environment, refer to the Monitoring a VMware vCloud
technical white paper.
6.2 Logging
This section describes logging architecture concerns and logging as a service. Logging requirements and
considerations for a public vCloud can be extensive. The requirements and considerations for a private vCloud
typically may not be as rigorous. For a private vCloud, refer to enterprise-specific requirements and guidelines.
Logs should be available in a vCloud for numerous reasons, including the following:
Regulatory Compliance. Collect logs to make them available for analysis, security review, compliance
requirements, as described in the Appendix: Security Considerations. Individual logs can then be used to
satisfy specific compliance controls; for example, a user access log can be used to verify access to a resource is
only by authorized users.
VMware vCloud
Customer Requirements. End customers (tenants) can retrieve logs that pertain to their environment in order
to satisfying their own requirements, many of which, such as compliance, will likely be similar to provider
requirements.
Operational Integrity. Operational alerts should be defined where specific logs trigger notifications for further
remediation. This will typically be a backup alert, secondary to monitoring.
Troubleshooting. Closely related to operational integrity, troubleshooting can be done with logs. For example,
the use of vShield Edge logs can show whether or not a specific external connection request is being passed
through the firewall or NATed by the firewall.
6.2.1 Logging Architectural Considerations
Redundancy
Many components rely on syslog for logging events. Syslog is a UDP-based protocol that lacks delivery
guarantees. To help ensure delivery:
Verify that infrastructure components have physically and logically redundant network interfaces.
Send logs to two syslog targets.
Where only a single syslog target is possible, it is recommended to log to a local syslog daemon configured to
retransmit the logs to two remote syslog targets. For example, vCloud Director 1.0 only supports a single
syslog target for its activity logs.
Where possible, place log receivers on DRS so vCenter will restart them in the case of failure.
Scalability
vCloud infrastructure components generate a relatively low level of logs for provider infrastructure. Customer
components, especially vShield Edge firewalls, can generate a very high volume of logs. Collecting logs on IOPS
performance is critical. Collecting logs on CPU performance is negligible, but will matter for analysis.
It is highly recommended that logs be collected to dedicated log partitions on collection servers.
Reporting
Logs need to be available to tenants. They should be able to download in raw format all vCloud Director and VSE
logs pertaining to their organizations/networks. Logs with customer identifiers should be flagged or indexed for
retrieval.
Customer activity in vCloud Director will generate logs that are flagged with their organization ID.
vShield Edge devices do not have unique identifiers in vCloud Director 1.0. Therefore, it is VMware recommends
you keep NAT-routed organization external networks and fenced vApps connected only to single-tenant
provider networks. When the vShield Edge device is deployed by vCloud Director and its external IP address is
allocated, the tenant can then be identified by the IP address.
VMware vCloud
The following is an example logging architecture.
vCenter
Log Collection Node
Log Collection Node
ESXi
Log Collection Node
ESXi
ESXi
DB
Org vDC
Inc.
LaaS
Inc.
Customer vApps
vCenter
Log Collection Node
ESXi
Log Collection Node
ESXi
Log Collection Node
ESXi
Web - Report/DL
Inc.
VMware,
VMware,
Inc.
VMware,
Inc.
VMware,
Inc.
VMware,
VMware,
Inc.
VMware,
Inc.
Inc.
VMware,
VMware,
Inc.
Collection/Agents - HA
Inc.
Customer vApps
VMware,
v
C
O
Inc.
CBM
Inc.
vSE
VMware,
vSM
Org vDC
VMware,
vCD
v
C
O
vCloud Resource Group Customer Resource Pools

LaaS
VMware,
vCloud Management Cluster
Inc.
VMware,
Inc.
VMware,
VMware,
Inc.
Log Reporting/Analysis Tier - Virtual/Dedicated
Customer Access
Downloads, Customer
Archiving
Figure 15. Architectural Example Drawing
6.2.2 Logging as a Service

Logging as a service can be done in two directions:
With customer collection and forwarding to provider servers for analysis and reporting.
With customer collection, reporting and analysis in the customer environment; and provider logs forwarded to
the customer environment.
Customer Collection forwarding to Provider
Pros:
Logs can be sent directly to collector even on customer private IP space.
Resources can be allocated at the customer level for collection, allowing more granular scaling of collection.
Cons:
More difficult to scale analysis; challenges correlating customer activity to storage consumption.
Collection node(s) still required even though utilization will be low.
Most of the resource consumption is on the storage and analysis side, so the resources billed via the IaaS model
will be minimal.
Customer Collection, with Provider Logs Forwarded Into Customer Environment
Pros:
Distributed analysis relies on general cloud resources and can scale.
Customer can employ their own analysis tools to organize and report on the data, or use a provider-supported
package or appliance.
Cons:
Provider needs duplicate copy of infrastructure logs for provider purposes.
Transmission of logs to the customer environment requires connectivity; either Internet or a provider service
network, and inbound traffic through a firewall into the customer environment, adding risks.
VMware vCloud
6.3 End-to-End Security Considerations with vCloud

Security in a vCloud can be considered for three areas: vCloud environment, user access, and network-level
workloads.
6.3.1 vCloud Environment Security
While vCloud Director is designed for secure multi-tenancy to reduce the impact of organizations on each
other additional steps can be taken to harden the environment. This is especially important in a service
provider environment where multiple organizations will be connected directly to the Internet.
Refer to the VMware vCloud Director Security Hardening Guide for detailed information on hardening VMware
vCloud Director. Refer also to the Appendix on Signed Certificates for guidance on setting up signed certificates
for vCloud Director.
6.3.2 User Access Security
Authentication and authorization mechanisms built into vCloud Director provide user security for vCloud
resources. Integration with a directory service (LDAPv3), such as Active Directory, OpenLDAP or Kerberos v5
can be configured. Refer to the VMware vCloud Director Administrators Guide for more information on how to
set up the naming services (LDAPv3), Active Directory, or OpenLDAP and Kerberos v5 integration.
It is up to the service provider to configure a directory service for vCloud Director. This is also true for a private
vCloud.
User access and privileges within vCloud Director is controlled through role-based access control (RBAC). Refer
to the VMware vCloud Director Administrators Guide for additional information on permissions, roles, and
default settings.
6.3.3 Securing Workloads at the NetworkLevel Workload Security
Network visibility (external or internal to an organization or vApp) and connection types (direct or NAT routed)
protect network-level workloads in a vCloud environment.
vCloud Director automatically deploys vShield Edge devices to facilitate routed network connections. vShield
Edge uses MAC encapsulation for NAT routing. This helps prevent Layer 2 network information from being seen
by other organizations in the environment. vShield also provides a firewall service that can be configured to
block inbound traffic to virtual machines connected to a public access organization network.
The Service Definition for Public Cloud, specifies how network options should be set for service providers for
security. Each of the organization networks are connected to the shared public network through a routed
connection.
A maximum of 8 public IP addresses should be allowed inbound access to an organizations virtual machines to
meet the requirements of the Service Definition. The organization administrator is the actual user that will be
responsible for making this configuration change. Once a vApp is created and virtual machines are added to it
and connected to the public access organization network, the vApp will obtain a private IP address from the
static IP pool previously established. The organization administrator can then configure the firewall and the NAT
external IP map for the newly created virtual machine, as well as the private IP address using the network
configure services wizard shown below.
VMware vCloud
Figure 16. Configure Firewall Services
Private vCloud network routing and firewall requirements depend on security policies, organizational
requirements, and workloads of the enterprise. For further details, see the Appendix on Security.
6.4 Workload Availability Considerations

vCloud Director works transparently with vCenter Server to provision and deploy virtual machines on hosts.
Therefore, it is imperative to architect redundancy and protect the infrastructure components. Provisioned
virtual machines can be protected by VMware HA. Virtual machines can also be protected using backup tools
within the Guest OS or vStorage API (VADP based) applications. See the Backup and Restore section for further
information.
At this time, virtual machines provisioned by vCloud Director cannot be protected by VMware FT or vCenter Site
Recovery Manager. See the Disaster Recovery section for further information.
6.4.1 Uptime SLAs at 99.99%
VMwares current Service Definition for a Public Cloud specifies a 99.9% uptime SLA. This may be sufficient for
noncritical applications or applications that are inherently highly available. Private and public cloud providers
have found that the market for an enterprise-ready cloud expects uptime SLAs at 99.99% (5 minutes per month)
or even greater.
This means for vCloud verification that:
End customers workloads are running.
End customer workloads are accessible (via the vCloud Portal and API, as well as through remote access
protocols).
VMware vCloud
To address a 99.99% uptime SLA, VMware can only control the resiliency of its vCloud platform components and
provide recommendations to mitigate single points of failure (SPOF) in the underlying infrastructure. A provider
can eliminate SPOF by ensuring redundancy, as listed below:
Redundant power sourced from multiple feeds, with multiple whips to racks, as well as sufficient backup
battery and generator capacity.
Redundant network components.
Redundant storage components.
Storage design needs to be able to handle the I/O load as well. Customer workloads may not be accessible
under high disk latency, file locks, and so forth.
Storage design should also be tied to business continuity and disaster recovery efforts, possibly including
array level backups.
Redundant server components (multiple independent power supplies, network interface cards (NICs), and, if
appropriate, host based adaptors (HBAs).
Sufficient compute resources for a minimum of N+1 redundancy within a vSphere HA cluster including
sufficient capacity for timely recovery.
Redundant databases and management
Appropriate change, incident, problem and capacity management processes must also be well defined and
enforced to make sure that poor operational processes do not result in unnecessary downtime. In addition to a
redundant infrastructure, employees or contractors responsible for operating and maintaining the environment
and the supporting infrastructure must be adequately trained and skilled.
A vCloud is capable of supporting an uptime SLA of 99.99% by following the guidelines in the table in Appendix
on vCloud Availability. The availability recommendations in the table allow a vCloud to achieve a 99.99% uptime
SLA but only with no SPOFs in the underlying infrastructure, required skills available, and suitable processes
defined and followed.
6.4.2 Load Balancing of vCloud Director Cells
vCloud Director (vCloud Director) cells are stateless front-end processors for the vCloud. Each cell has a variety
of purposes and self-manages various functions among cells, while connecting to a central database. The cell
manages connectivity to the cloud and provides both API and UI end-points, or clients.
Multiple cells (a load balanced group) should be used to address availability and scale. This is typically achieved
by load balancing or content switching this front-end layer. Load balancers present a consistent address for
services regardless of the underlying node responding. They can spread session load across cells, and monitor
cell health and add/remove cells from the active service pool. The group should not be considered a true cluster
since there is no failover from one cell to another.
In general, any load balancer that supports SSL session persistence and has network connectivity to the public
facing Internet or internal service network can perform load balancing of vCloud Director cells. General concerns
around performance, security, manageability, and so forth should be taken into account when deciding to share
or dedicate load balancing resources.
For the purposes of this reference architecture, the load balancer is assumed to be a dedicated virtual machine
or hardware device.
For additional information on load balancing, see the Appendix on vCloud Availability, and the section on load
balancers.
VMware vCloud
6.4.3 I/O Considerations

vCloud Director offers controls for new organizations to guard against the misuse of resources by other
organizations. These include:
Quotas for running and stored virtual machines.
Determines how many virtual machines each user in the organization can store and power on in the
organizations virtual datacenters. The quotas you specify act as the default for all new users added to
the organization.
Limits for resource intensive operations.
Prevents resource intensive operations from affecting all the users in an organization and also provide a
defense against denial-of-service attacks.
Simultaneous VMware Remote Console (VMRC) connections.
Limits the number of simultaneous connections for performance or security reasons.
NOTE: VMware currently does not recommend the use of SIOC/NIOC in vSphere beneath the cloud abstraction
layer. vSphere contains options for storage I/O control (SIOC) and network I/O control (NIOC) but these
functions are not integrated into vCloud Director and their use in vCloud could cause unpredictable results.
6.4.4 Disaster Recovery
Disaster Recovery (DR) focuses on the recovery of systems and infrastructure after an incident that interrupts
normal operations. A disaster can be defined as partial or complete unavailability of resources and services,
including software, the virtualization layer, the cloud layer, and the workloads running in the resource groups.
There are different approaches and technologies supported, but there are at least two areas that require disaster
recovery: the management cluster and consumer resources. Consumer resources are described later in this
document. There are different approaches and technologies supported. The two areas include the management
cluster and the consumer resources, as defined later.
Management Cluster Disaster Recovery
Good practices at the infrastructure level will lead to easier disaster recovery of the management cluster. This includes
technologies such as HA and DRS for reactive and proactive protection at the primary site. vCenter Heartbeat can
also be used to protect vCenter Server, specifically, at the primary site. For multi-site protection, vCenter Site
Recovery Manager protection of virtual machines is VMwares solution that works normally for this use case, since the
management VMs are not part of a cloud instance of any type (but, rather, running the cloud instances).
Cloud Consumer Resources Disaster Recovery
This section focuses on disaster recovery of the cloud infrastructure to handle failure to an alternate site. vCenter
Site Recovery Manager is not supported, although there are manual steps that can be applied as long as vApp
metadata is saved, configuration information is matched between the primary site and the recovery site, and the
documented steps are validated.
While Site Recovery Manager is vCenter Server-aware, vCloud Director is not Site Recovery Manager-aware.
Without the collaboration between vCloud Director and Site Recovery Manager, the mechanisms working
beneath the covers to synchronize VMs cannot work to keep vCloud Director in sync as well, and as a result the
recovery of vCloud Director can be problematic. While it is possible to architect a solution where one sites total
environment, 100% of the operational parameters of that site including IP addressing, start-up order of
dependent systems and the like, are duplicated to another site to recover to, it would be very difficult to
implement and maintain, so is therefore out of scope for this document.
VMware is actively working on a streamlined solution to this situation, to be addressed by future product enhancements.
6.4.5 Backup and Restore of vApps
This section focuses on handling of backup/restore procedures with the vApps that are deployed into the cloud.
Traditional backup tools do not capture the required metadata associated with a vApp, including owner,
network, and organization. This results in recovery/restoration issues. Without this data, recovery must include
manual steps and configuration attributes to be manually re-entered.
VMware vCloud
Within a vCloud environment, a vApp can be a single virtual machine or group of virtual machines, treated as
one object. Backup of vApps on isolated networks must be supported. Identifying inventories of individual
organizations becomes challenging based on current methods that enumerate the backup items using vSphere,
which uses UUIDs to differentiate objects whereas vCloud Director uses object IDs.
For backing up and restoring vApps, VMware recommends the use of vStorage API Data Protection (VADP)
based backup technologies. This technology has no agents on Guest OSs, is centralized for improved
manageability, and has a reduced dependency on backup windows.
Guest-based backup solutions may not work in a vCloud because not all virtual machines are accessible by
network. Also, virtual machines may have identical IP addresses that can cause problems. Therefore, backups of
vCloud vApps must require a virtual machine-level approach.
When deploying virtual machines (as part of a vApp), use the full name and computer name fields to specify
realistic names that will help describe the virtual machines. If this is not done, the generic information in these
fields can make it hard to specify individual virtual machines. vApps and virtual machines that are provisioned by
vCloud Director have a large GUID template_name,which means many virtual machines could appear to be very
similar and make it hard for a user or administrator to ask for a specific virtual machine to be restored.
VMware Solutions
VMware Data Recovery is a vStorage API Data Protection based solution from VMware. Other vStorage API Data
Protection based backup technologies are available from third-party backup vendors. Currently due to the UUID
versus object ID issue discussed above, VMware Data Recovery cannot be used with VMware vCloud Director.
Backup of vCloud workloads has a few requirements to address. VMware recommends that clients validate the
level of support provided by the vendor to make sure client requirements are supported. Here is a list of Cloud
vApp requirements to ask your vendor about:
vA p p R e q u i r e m e n t
vStorage API Data

Protection (VADP)
integration
D e ta i l
vCloud Director integration
vApp requirements
vStorage API Data Protection provides change-block tracking

capability to reduce backup windows.
Integration to enable backup of isolated VMs and vApps.
Integration with vStorage API Data Protection to provide
LAN-free and server-free backups to support better
consolidation rations for vCloud and the underlying vSphere
infrastructure.
Use of the virtual machine UUID versus virtual machine name will
support multi-tenancy and avoid potential name space conflicts.
Interface support for cloud provider administrator teams. In the
future, consumer (organization administrator and users) access
will potentially be provided by some vendors.
Include vCloud metadata for the vApps. This includes temporary
and permanent metadata per VM/vApp. This is required to make
sure that recovery of the VM/vApp will have all data required to
support resource requirements and SLAs.
Provide vApp granularity for backups. Support backup of multitiered vApps (for example, a Microsoft Exchange vApp that has
multiple virtual machines included. Backup selection of the
Exchange vApp would pick up all the underlying virtual
machines that are part of the main vApp).This is something that
is not available today, but is being developed by vendors.
Table 6. vCloud vApp Requirements Checklist
VMware vCloud
Challenges
The following is a list of backup and restore challenges:
vApp naming posing conflict issues between tenants
vApp meta data required for recovery
Multi-object vApp backup (protection groups for multi-tiered vApps)
Manual recovery steps in the cloud
Support for backup of vApps on isolated networks or with no network connectivity
Enumeration of vApps by organization for use by the organization administrator
Enumeration of vApps by organization and provider for use by the organization provider
User initiated backup/recovery
Support of provider (provider administrator) and consumer (organization administrator and user)
7. Sizing the vCloud

Sizing the management cluster is fairly predictable, with the main variables being the number of vCloud Director
cells and the size of the vCloud Director database.
Sizing guidelines for the management cluster were provided earlier in the section on establishing provider virtual
datacenters.
The vCloud consumer resources have unpredictable usage, and thus should be sized by making an estimate of
the initial capacity required, and employing capacity management techniques. Capacity management
techniques predict future usage needs based upon past usage trends.
7.1 Initial Sizing of Cloud Consumer Resources

Sizing a vCloud is dependent on the service definitions for that vCloud. For a private vCloud, the service
definition may not specifically call out a required number of workloads to support. In that case, you may want to
take into consideration how a public vCloud is initially sized.
For a public vCloud, initial sizing for the cloud consumer resources can be difficult to predict since the provider is
not in charge of what the consumers may run. The provider is also not aware of existing usage statistics for
virtual machines that are run in the vCloud. The information below should assist in initial sizing of the vCloud
environment and is based on information from the Service Definition for a Public Cloud. This information is being
provided as examples. VMware recommends that you engage your local VMware representative for detailed
sizing of your environment.
The Service Definition states that 50% of the total number of virtual machines will be run in the reservation pool
model and 50% will be run in the Pay-As-You-Go model. Furthermore, the reservation pool is split into small,
medium, and large pools with a respective split of 75%, 20%, and 5%. Using the 50% above, this means that small
represents 37.5% of the total, medium represents 10% of the total, and large represents 2.5% of the total number
of virtual machines in the environment.
The definition for these resource pools and the split with the virtual machines is listed below. The total number of
virtual machines of 1,500 from the Service Definition for a Public Cloud is used in this example. You can change
this total to reflect your own target virtual machine count.
VMware vCloud
T y p e o f R e s o u rc e P o o l
Tota l P e rc e n tag e
Tota l V M s
Pay-As-You-Go
50%
750
Small Reservation Pool
37.5%
563*
Medium Reservation Pool
10%
150
Large Reservation Pool
2.5%
37*
TOTAL
100%
1,500
* Note: Some total virtual machines are rounded up or down due to percentages.
Table 7. Definition of Resource Pool and Virtual Machine Split
The Service Definition for a Public Cloud also calls out the distribution for virtual machines in the environment
with 45% small, 35% medium, 15% large, and 5% extra large. Below is a table that shows the total amount of
memory, CPU, storage, and networking based on the assumptions and the total virtual machine count from the
Service Definition for a Public Cloud.
It e m
P e rc e n t
v C PU s
Memory
Sto rag e
N e tw o r k i n g
Small
45%
675
675 GB
40.5 TB
400 GB
Medium
35%
1,050
1,050 GB
31.5 TB
300 GB
Large
15%
900
900 GB
54 TB
400 GB
Extra Large
5%
600
600 GB
4.5 TB
200 GB
TOTAL
(1,500)
100%
3,225
3,225 GB
130.5
1,300 GB
Table 8. Memory, CPU, Storage, and Networking
The above numbers may shock you. Before you determine your final sizing numbers, you should refer to VMware
best practices for common consolidation ratios on the above resources. An example table below shows what
final numbers could look like using typical consolidation ratios seen in field deployments.
R e s o u rc e
Before
R at i o
After
CPU
3,225
8:1
403 vCPUs
Memory
3,225 GB
1.6:1
2,016 GB
Storage
130.5 TB
2.5:1
52 TB
Network
1,300 GB
6:1
217 GB
Table 9. Example Consolidation Ratios
The above calculations could be served by 16 of the following hosts.

Socket count: 4
Core count: 6
Hyper threading: Yes
Memory: 128 GB
Networking: Dual 10 GigE
VMware vCloud
The above calculations do not take into account the storage consumed by consumers or providers templates.
The above calculations also do not take into account the resources consumed by the vShield Edge appliances
that are deployed for each organization. There will be a vShield Edge for each private organization network and
external organization network. Given the current Service Definition target of 25 organizations, a maximum of 275
vShield Edge appliances will be created.
The specifications for each vShield Edge appliance are listed below.
CPU: 1 vCPU
Memory: 64 MB
Storage: 16 MB
Network: 1 GigE (this is already calculated in the throughput of the workloads and should not be added again)
7.2 Capacity Management

One of the key benefits of implementing a vCloud is the ability for service provider customers (here equivalent
to public cloud providers or private cloud internal IT) to rapidly provision vApps into the vCloud environment.
The goal of capacity management is to make sure that sufficient capacity exists within the vCloud infrastructure
to meet the current and future needs of the service provider customers under normal circumstances. Sufficient
reserve capacity must be maintained within the vCloud infrastructure to prevent vApps from contending for
resources, and thus potentially breaching services levels agreed upon for customers.
As vApps are provisioned and consumed within the vCloud infrastructure, capacity is reduced and additional
capacity must be procured and provisioned. Capacity management processes should be instituted to make sure
appropriate resources are available to support the service level requirements associated with vApp provisioning
and performance. Proper capacity management will also prevent costly over-provisioning of hardware resources
by balancing high resource utilization with agreed-upon levels of performance.
As the vCloud is consumed, additional capacity must be added to the cloud consumer resources to allow for
anticipated future demand while preserving sufficient headroom. To predict future capacity needs, youll need to
analyze current capacity usage and trends to determine growth rates as well as estimate future needs, largely
coming from new consumers and projects.
VMware vCenter CapacityIQ is a tool that can be used to monitor and predict capacity usage and requirements.
In a vCloud environment, while CapacityIQ can provide details on capacity at the virtual machine and host levels,
at this time it does not provide insight at the provider and organization virtual datacenter levels.
For further information on capacity planning and management, refer to the VMware vCenter CapacityIQ
Installation Guide.
Alternatively, refer to the Capacity Planning Appendix for guidance on how to manually calculate capacity
requirements and forecast capacity.
VMware vCloud
8. Implementing Your vCloud

Use the documents in the vCloud Reference Architecture Kit to help you implement a vCloud. Implementation
Examples are provided for a public and private vCloud and help you visualize the end result of applying this
Architecting a vCloud guide with the appropriate Service Definitions.
What is a vCloud?
1
Requirements for a vCloud
What should the vCloud offer?
What to consider in building a vCloud
2a
Service Definition for a
Public Cloud
Public Cloud
(Service Provider)
Reference Implementation Examples

4a
Public vCloud
Implementation Example
3
2b
4b
Service Definition for a
Private Cloud
Private Cloud
Business requirements for the vCloud.

Public cloud doc draws upon VMware
vCloudDatacenter program. Private Cloud
doc shows typical use case.
Architect-level guide identifying what

components go into a vCloud and design
considerations. Uses Service Definition as
input into what to design.
Private vCloud
Implementation Example
Use as a reference for what a
vCloud architecture design
document can/should look like.
Audience: vCloud Architect familiar with vSphere best practices (ideally VCP-level) and exposure to vCloud product components
Figure 17. Reference Architecture Kit
VMware vCloud
9. Appendix: vCloud Director Cell Monitoring

The following table represents a subset of MBeans that can be used for improving the monitoring performance
of a vCloud instance.
Lo ca l u s e r s e s s i o n s
Mbean
com.vmware.vcloud.diagnostics.UserSessions
Description
Local (cell) user session statistics
Cardinality
Instance ID
n/a
Attribute
Description
totalSessions
Total number of sessions created on this cell
successfulLogins
Total number of successful logins to this cell
failedLogins
Total number of failed login requests to this cell
G lo ba l u s e r s e s s i o n s
Mbean
com.vmware.vcloud.GlobalUserSessionStatistics
Description
List of active user sessions by organization.
Cardinality
Instance ID
n/a
Attribute
Description
organization
Database ID of the organization
active
Number of active sessions
Open_Session
number of open sessions
Data acc e s s d i ag n o s t i c s
Mbean
com.vmware.vcloud.diagnostics.DataAccess
Description
Cardinality
Instance ID
Conversation
Attribute
Description
lastAccessInfo.objectType
object type of the last database object accessed
lastAccessInfo.accessTime
time taken to access the last database object accessed
worstAccessInfo.objectType
object type of the worst (slowest) database object access
worstAccessInfo.accessTime
time taken by the worst (slowest) database object access
Databa s e C o n n e ct i o n P o o l
Mbean
com.vmware.vcloud.datasource.globalDataSource
Description
Statistics and configuration information about the database

connection pool. This information is currently specific to the
database JDBC driver being used (Oracle).
Cardinality
VMware vCloud
Instance ID
Attribute
Description
abandonedConnectionTimeout
availableConnectionsCount
borrowedConnectionsCount
connectionHarvestMaxCount
connectionHarvestTriggerCount
connectionPoolName
connectionWaitTImeout
databaseName
database connection database name (SID)
dataSourceName
fastConnectionFailoverEnabled
inactiveConnectionTimeout
initialPoolSize
loginTImeout
maxConnectionReuseCount
maxIdleTime
maxPoolSize
maximum number of connections allowed in the pool
maxStatements
minPoolSize
minimum number of connections that will exist in the pool
networkProtocol
network protocol used by JDBC driver
ONSConfiguration
portNumber
database conenction port number
SQLForValidateConnection
timeoutCheckInterval
timeToLiveConnectionTimeout
URL
database connection URL
user
database connection username
validateConnectionOnBorrow
V I M O p e rat i o n s
Mbean
com.vmware.vcloud.diagnostics.VlsiOperations
Description
Cardinality
1 per VIM end-point (VC or host agent)
Instance ID
VIM end-point URL
Attribute
Description
ObjectType.MethodName.httpTime
the total network round-trip time taken to make the MethodName

call on object of type ObjectType in the VIM end-point.
VMware vCloud
Pr e s e n tat i o n A PI M e t h o d s
Mbean
Description
Cardinality
1 per presentation layer method
Instance ID
method name
Attribute
Description
currentInvocations
currently active invocations
totalFailed
total number of failed executions
totalInvocations
total number of invocations over time
executionTime
total time taken to execute
J e tt y
Mbean
com.vmware.vcloud.diagnostics.Jetty
Description
Web server request statistics
Cardinality
2, 1 for REST API and 1 for UI
Instance ID
UI Requests for UI, REST API Requests for REST API
Attribute
Description
Active
number of web requests currently being handled
R ES T A PI
Mbean
Description
Cardinality
1 per operation stage/granularity: RoundTrip, BasicLogin, Logout,

Authentication, SecurityFilter, ConversationFilter, JAXRSServlet.
RoundTrip is the most interesting, as it represents the overall REST
API performance.
Instance ID
One of: RoundTrip, BasicLogin, Logout, Authentication,

SecurityFilter, ConversationFilter, JAXRSServlet
Attribute
Description
currentInvocations
totalFailed
totalInvocations
executionTime
Ta s k E x e c u t i o n
Mbean
com.vmware.vcloud.diagnostics.TaskExecutionJobs
Description
Statistics about long running tasks
Cardinality
1 per task
Instance ID
Name of task
Attribute
Description
currentInvocations

VMware vCloud
totalFailed
totalInvocations
executionTime
Q u e r y S e rv i c e ( UI )
Mbean
com.vmware.vcloud.diagnostics.QueryService
Description
Presentation layer query service statistics
Cardinality
1 per query
Instance ID
query name
Attribute
Description
currentInvocations
totalFailed
totalInvocations
executionTime
returnedItems
number of items returned by successful query executions
VC Ta s k M a n ag e r
Mbean
com.vmware.vcloud.diagnostics.VcTasks
Description
VC task management statistics
Cardinality
Instance ID
Attribute
Description
successfulTasksCount
total successful tasks
failedTasksCount
total failed tasks
waitForTaskInvocationsCount
total invocations of VIM wait for task
completedWaitForTasksCount
total completed task waits
historicalTasksCount
total historical task updates received
vcRetrievedTaskCompletionsCount
total task completions received
taskCompletionMessagesPublishedCount
total task completion messages published on message bus
taskCompletionMessagesReceivedCount
total task completion messages received on message bus
success_elapsedTaskWaitTime
time elapsed for successful tasks
failed_elapsedTaskWaitTIme
time elapsed for failed tasks
V I M I n v e n to r y U p dat e Pr o c e s s i n g - o bj e ct u p dat e s tat i s t i c s
Mbean
com.vmware.vcloud.diagnostics.VimInventoryUpdates
Description
Inventory processing statistics
Cardinality
3, one for ObjectUpdate, PropertyCollector and UpdateSets

respectively
Instance ID
ObjectUpdate
Attribute
Description
VMware vCloud
totalUpdates
total number of object updates received
totalFailed
total number of object updates failed to be processed
executionTime
time taken for updates
V I M I n v e n to r y Ev e n t s
Mbean
com.vmware.vcloud.diagnostics.VimInventoryEvents
Description
VIM inventory event manager statistics. Tracks the frequency of

common vCenter events.
Cardinality
1 per folder per VC URL, 1 MBean per event name
Instance ID
event name
Attribute
Description
totalInvocations
totalFailed
total number of VIM inventory events that were failed to be handled
executionTime
total time time to handle VIM inventory events
VC Obj e ct Va l i dat i o n s
Mbean
com.vmware.vcloud.diagnostics.VcValidation
Description
VC object validation statistics
Cardinality
1 global plus 1 per validator
Instance ID
null = global, validator name = per validator
Attribute
Description
totalInvocations
total number of validation executions
executionTime
total time spent in validator
totalItemsInQueue
total items currently queued for validation (global)
objectsInQueue
total items currently queued for validation (per validator)
objectBusyRequeueCount
total number of objects requeued for validation due to object being busy
loadValidationObjectTime
time taken to load validation object
duplicatesDiscarded
total number of discarded duplicate validations
VC Obj e ct Va l i dat i o n R e act i o n s
Mbean
com.vmware.vcloud.diagnostics.Reactions
Description
validation reaction statistics
Cardinality
1 global plus 1 per reaction
Instance ID
null = global, reaction name = per reaction
Attribute
Description
totalReactionsFired
total number of reaction executions
requeueCount
total number of reactions requeued due to objects being busy
totalInvocations
total number of executions of this reaction
executionTime
total time spent in reaction
VMware vCloud
failedReactions
total number of failed reactions
objectRequeueCount
number of times this reaction was requeued due to objects being

busy
VC c o n n e ct i o n s
Mbean
com.vmware.vcloud.diagnostics.VimConnection
Description
Cardinality
1 per VC
Instance ID
VC-VcInstanceId where VcInstanceId is an integer identifying the

virtual center
Attribute
Description
Connected Count
Total successful connections
Disconnected Count
Total disconnections
Start Count
Total number of times the VC listener was started
UI Vim Reconnect Count
Total number of times the VC was reconnected through the UI
Act i v e M Q
Mbean
com.vmware.vcloud.diagnostics.ActiveMQ
Description
Active MQ (message bus) statistics
Cardinality
1 global and 1 per peer VCSD cell (each cell other than the current
one)
Instance ID
Global = global statisticsto_cellName_cellPrimaryIp_

cellUUID=per cell
Attribute
Description
lastHealthCheckDate
last time health check was performed
messageRoundTripDurationMs
Time taken for an echo message to be sent and returned
unreachableCells
Total number of unreachable cells
isHealthy
Health of connections to pear
reachableCells
Total number of reachable cells
T ra n s f e r S e rv e r
Mbean
com.vmware.vcloud.diagnostics.TransferService
Description
Transfer server statistics
Cardinality
Instance ID
Attribute
Description
successfulPuts
number of items successfully transferred
failedPuts
number of items that were failed to be transferred
successfulUploads
number of successful upload operations
Table 10. MBeans Used To Monitor vCloud Cells
VMware vCloud
10. Appendix: vCloud Availability Considerations

A vCloud is capable of supporting an uptime Service Level Agreement (SLA) of 99.99% by following the
guidelines presented in the following table. Following the availability recommendations in the table allows a
vCloud to be capable of supporting a 99.99% uptime SLA but only if there are no single points of failure (SPOF)
in the underlying infrastructure, the required skills are available, and suitable ITIL processes are defined and
adhered to.
M a i n ta i n i n g R u n n i n g W o r k loa d
Component
Availability
Failure Impact
vSphere ESXi hosts
All VMware ESXi hosts will be configured

in highly available clusters with a
minimum of n+1 redundancy. This will
provide protection not only for the
customers virtual machines, but also the
virtual machines hosting the platform
portal/management applications and all
of the vShield Edge appliances.
In the event of a failure of a host,

VMware HA will detect the failure within
13 seconds and commence powering on
the failed virtual machines on other
hosts within the cluster. (Analogous to
pressing the power button on a physical
server, but not including time to boot
the OS or launch applications)
VMware HA Admission Control ensures
sufficient resources are available to
restart the virtual machines. The
admission control policy Percentage of
cluster resources is recommended as
it is flexible while guaranteeing resource
availability.
The following whitepaper contains best
practices around increasing availability /
resiliency: http://www.vmware.com/
files/pdf/techpaper/VMW-Server-WPBestPractices.pdf
It is also recommended that vCenter is
configured to proactively migrate virtual
machines off a host in the even the
hosts health become unstable. Rules
can be defined in vCenter when
monitoring host system health.
Virtual machine
resource
consumption
VMware DRS will automatically migrate

virtual machines between hosts to make
sure the cluster is balanced to reduce
the risk of a noisy neighbor virtual
machine monopolizing CPU and
memory resources within a host at the
expense of other virtual machines
running on the same host.
No impact. Virtual machines are

automatically migrated between hosts
with no downtime by VMware DRS.
VMware Storage I/O Control will

automatically throttle hosts and virtual
machines when it detects that the
datastore is congested/bottlenecked.
This ensures that a noisy neighbor
virtual machine does not monopolize
storage I/O resources. Storage I/O
Control ensures each virtual machine will
receive the resources it is entitled to by
leveraging the shares mechanism.
No impact. Virtual machines and ESXi

hosts are throttled by Storage I/O
Control automatically based on their
entitlement relative to the amount of
shares or the maximum amount of IOPS
configured. For more information on
Storage I/O Control, check out the
whitepaper: http://www.vmware.com/
files/pdf/techpaper/VMW-vSphere41SIOC.pdf
VMware vCloud
vSphere ESXi host

network
connectivity
ESXi hosts will be configured with a

minimum of 2 physical paths to each
required network (port group) to make
sure a single link failure does not impact
platform or virtual machine connectivity
this should include management and
vMotion networks. The Load Based
Teaming mechanism will be used to
avoid oversubscribed network links.
No impact. Failover will occur with no

interruption to service. Configuration of
failover and failback as well as
corresponding physical settings such as
Portfast are a requirement.
vSphere ESXi host

storage
connectivity
ESXi hosts will be configured with a

minimum of 2 physical paths to each
LUN or NFS share to make sure a single
storage path failure does not result in an
impact to service. Path Selection Plug-in
will be selected based on storage
vendors best practices.
No impact. Failover will occur with no

interruption to service.
M a i n ta i n i n g W o r k loa d Acc e s s i b i l i t y
Component
Availability
Failure Impact
VMware vCenter
Server
vCenter Server will run as a virtual

machine and make use of vCenter
Server Heartbeat.
vCenter Server Heartbeat provides a

clustered solution for vCenter Server
with fully automated failover between
nodes, thereby providing near zero
downtime.
VMware vCenter
Database
VMware vCenter Database resiliency is

provided with vCenter Heartbeat if MS
SQL is used or Oracle RAC if Oracle is
used.
vCenter Heartbeat or Oracle RAC

provides a clustered solution for a
vCenter database with fully automated
failover between nodes, thereby
providing zero downtime.
vCloud component
databases (vCloud
Director and
Chargeback)
VMware vCloud component database

resiliency is provided with Oracle RAC.
Oracle RAC supports the resiliency of

the vCloud Director and Chargeback
databases as it maintains vCloud
Director state information and the
critical Chargeback data required for
customer billing respectively. While not
required for maintaining workload
accessibility, clustering the Chargeback
database makes sure providers can
accurately produce customer billing
information.
VMware vCenter
Chargeback
vCenter Chargeback Data Collectors

vCenter Server, vCloud Director, vShield
Manager shall be distributed in the
event of failure
In the event that one of the data

collectors for a group should go offline,
the others will pick up the load such that
transactions are captured by vCenter
Chargeback.
vC lo u d I n f ra s tr u ct u r e Pr ot e ct i o n
Component
Availability
Failure Impact
VMware vCloud
vShield Manager
vShield Manager will receive the

additional protection of VMware FT,
resulting in seamless failover between
hosts in the event of a host failure.
VM Monitoring is enabled on a cluster
level within HA and uses the VMware
Tools heartbeat to verify vShield
Manager is alive. When a virtual machine
fails and thus the VMware Tools
heartbeat is not updated VM Monitoring
will verify if any storage or networking
I/O has occurred over the last 120
seconds before the virtual machine will
be restarted.
Infrastructure availability yes, service

availability no. vShield Edge devices will
continue to run without the
management control, but no addition
edge appliances or modifications to
existing can occur until the service
comes back online.
It is also recommended to create a

scheduled backup of vShield Manager to
an external FTP or SFTP server.
vCenter
Chargeback
vCloud Director
vShield Edge
vCenter Chargeback virtual machines

will be deployed as a two node, load
balanced cluster. Multiple Chargeback
data collectors can be deployed remotely
to avoid a single point of failure.
There is no impact on Infrastructure

availability or customer virtual machines.
The vCloud Director virtual machines

will be deployed as a load balanced,
highly available clustered pair in an
N+1 redundancy set up, with the option
to scale out when the environment
requires this.
Session state of users connected via the

portal to failed instance will be lost. They
will be able to reconnect immediately.
vShield Edge can be deployed through

API and vCloud Director. To provide
network reliability VM Monitoring will be
enabled. In case of a vShield Edge
Guest OS failure VM Monitoring will
restart the vShield Edge device. vShield
Edge appliances do not have VMware
Tools and thus are not monitored as part
of VMware HA guest OS monitoring.
Partial temporary loss of service. vShield

Edge is possible connection into
organization.
While not required for maintaining

workload accessibility, clustering the
vCenter Chargeback servers makes sure
providers can accurately produce customer
billing information and usage reports.
No impact to customer virtual machines.
No impact to customer virtual machines

or VM Remote Console (VMRC) access.
All external network routed connectivity
will be lost if a vShield Edge appliance
Table 11. vCloud Availability Considerations
VMware vCloud
11. Appendix: Security Considerations

11.1 Network Access Security
A VPN can be used to securely connect multiple vCloud deployments. vShield Edges VPN functionality allows
the creation of site-to-site tunnels using IPSEC. It supports NAT-T traversal for using IPSEC through Network
Address Translation (NAT) devices. Typical use cases include the following:
C at e g o r y
Multi-site cloud deployment
vShield VPN can connect multiple cloud deployments. For example, an

organizations virtual datacenter at a service provider on the West Coast could
be connected with the organizations virtual datacenter at a service provider on
the East Coast.
NOTE: Each internal subnet must have a unique address space to connect
successfully.
Because vShield also provides address translation, it is possible to deploy
multiple organization virtual datacenters at different providers using the same
RFC1918 address space. Unique subnets are required to connect them.
Office to cloud VPN
It is possible to have a permanent VPN from a router-based VPN, such as a

Cisco 2821 Integrated Services Router, to a cloud environment with the vShield
Edge. This can also be accomplished via a Linux-based gateway since vShield
VPN is compatible with Openswan, a Linux IPSEC implementation.
Client to cloud VPN
Client software is generally not supported, although robust clients with static
IPs that support pre-shared key authentication can connect.
Table 12. Network Access Security Use Cases
To configure vShield VPN, the endpoint connecting to the vShield Edge device must:
Support the following:
IKE for ISAKMP/Oakley
Pre-Shared Key Authentication Mode
3DES or AES128 encryption
SHA1 authentication
Diffie-Helman Group 2/5 (1024 bit/1536 bit, respectively)
PFS (Perfect Forward Secrecy)
ESP Tunnel Mode
Disable ISAKMP aggressive mode
11.2 Compliance
Audit concepts applied to a cloud environment, such as segmentation and monitoring, reveal new challenges.
Elasticity may break old segmentation controls and the ability to isolate sensitive data within a rapidly growing
environment. Role-based access controls and virtual firewalls must also demonstrate compatibility with audit
requirements for segmentation, including detailed audit trails and logs. Can a provider guarantee that an off-line
image with sensitive data in memory is accessible only by authorized users, and can a log tell who accessed it
and when? Multiple admin-level roles are necessary for cloud resource management.
The complexity of cloud environments, coupled with new and different technology, requires careful audits to
document and detail compliance. The following are a set of common audit concerns within the cloud.
VMware vCloud
Concern
D e ta i l
Hypervisor
An additional layer of technology is present in every cloud and therefore

presents a different attack surface. It introduces a layer between the
traditional processing environment and the physical layer, which brings
vulnerabilities of its own as well as new paths of attack related to its
communication with the layers above and below.
Segmentation and isolation
May expose sensitive data when not configured and monitored properly;
physical and logical isolation has always been an audit concern. The ease
and speed of change to a virtualized environment within cloud
computing, often called elasticity, makes the setup of review of
segmentation controls even more critical to compliance.
Different/multiple primary
functions per host
A cloud can make much more efficient use of hardware, but this brings
auditors to assess whether sensitive data is at risk just from the proximity
of other virtual systems managed to a lesser level of security. Some
compliance standards for example, require one primary function per
server (or virtual server), as illustrated below.
App App App
App
Regulated
Data
App App
FW, IDS, AV
FW, IDS AV
FW, IDS, AV FW, IDS, AV
ESXi
ESXi
Server
Server
Server
Routers
Switches
Firewalls Load Balancers
Management
Controls
App App
Nonregulated Data
Distributed Network
Hypervisor
ESXi
ESXi
Cluster
Storage Fabric
Physical
Storage
Server
Enforcement of least privilege
In a cloud environment, remote network access becomes the only path

for customers to manage an environment. Before, physical access was
audited for equipment installation and modification, now software
controls customer access capabilities. Authorization software has to be
more sophisticated to handle every user, group, and role request for a
cloud customer.
Machine state and migration
The ability of systems to quickly change and move within the cloud gives
auditors a need to track this.
Data is much less permanent
Cloud environments make extensive use of short-lived instances. Virtual

machines may have a lifecycle far shorter than physical systems because
they are so easy to provision and then repurpose. The systems often
share data across large arrays. Permanence of data is also affected by
environments that push as much storage as possible through high-speed
memory to avoid the latency of spinning disk.
Immaturity of monitoring solutions

in cloud environments
Customers need a view of their audit trails that is unique to their own use
of the cloud environment and that can be used for investigations.
Providers must enhance the sophistication of existing log tools in order to
keep up with the new technology and new management practices within
a cloud environment.
Table 13. Audit Concerns Within The Cloud
VMware vCloud
11.3 Use Cases: Why Logs Should be Available

Recording and monitoring are important to mitigate damage and prevent future attacks. An audit log enables
the organization to verify compliance, detect violations, and initiate remediation activities. It can help detect
attempts, whether successful or not, for unauthorized access, information probes, or disruption.
It is a best practice to regularly examine logs for suspicious, unusual, or unauthorized activity. External laws and
regulations also require specific levels of monitoring and verification for access and authorization. Rules are
necessary to restrict access, while routine log analysis also helps identify system configuration errors, failures,
and enforcement of adherence to SLAs.
Logs serve an essential purpose and are a foundation for most controls in a regulation; they track and record
changes and incidents to form an audit trail. Log purposes include:
Compliance requirements. Logs are required for all compliance regulations to assist with control auditing as
well as breach review, analysis and response. Individual logs often satisfy specific compliance controls; for
example, an authentication log can verify a resource was accessed by only authorized users.
Customer requirements. End customers (tenants) can retrieve logs that pertain to their environment in order to
meet their own requirements.
Operational integrity. Operational alerts should be defined for logs to trigger notifications for remediation.
This may be a backup alert, secondary to monitoring.
Troubleshooting. Closely related to operational integrity, logs are essential for troubleshooting. The use of
vShield Edge logs, for example, can show whether a specific external connection request is being passed
through or NATed by the firewall.
The following list is the minimum set of data types required to adequately log cloud environment activity for
regulatory compliance:
User (including system account) access
Action taken
Use of identification and authentication mechanisms
Start and stop of audit logs
Creation or deletion of system-level objects
The audit trail entries recorded for each event must include the following six details:
Identification (ID)
Type of event
Date and time
Success or failure
Origination of event
ID of affected data or component
The logs must be reviewed at least daily for all system components, and especially systems that handle intrusion
detection, authentication, and authorization. Whereas daily review of logs may not be sufficient on its own to
detect incidents, they also must be retained for a period consistent with effective use and legal regulations.
The laws for log retention range from one year to more than twenty. Therefore, log archives should always be
able to provide at least one year of history, typically scheduled to match financial calendar cycles, and a
minimum of three months available for immediate response and review in case of an incident.
VMware vCloud
11.3.1 Example Compliance Use Cases for Logs

The following use cases are a sample of events that benefit from careful logging and monitoring in the cloud
environment. Other examples may include unauthorized services or protocols, remote login success and
certificate changes.
Shared accounts. An investigation is initiated to review network outages and finds multiple instances of an
Administrator account had logged into critical servers before failure. Shared accounts make it very difficult to
trace fault to one individual; it is impossible to determine from the logs on that system which person was
logged into the user account that made the error. Therefore, usage must be tied to an individual user ID and
unique password with correct time to aid in investigations. Systems also should be configured to detect any
and all use of generic IDs such as an administrator or root account and trace them to unique identities.
User account changes. A malicious user finds an unpatched flaw in an environment that allows elevation of
privileges. That user then uses system-level privileges to create a new bogus user object from which to launch
further attacks. A user object is a Microsoft Widows Domain or Local user account, for example. User object
logs can be used to figure out when a name was changed or an account added. This assists in detection of
actions without authorization or users trying to hide attacks.
Unauthorized software. Malware or a new virtual machine instance in the cloud, can be found in system object
logs. A system must track system objects that are added, removed or modified. This can be very helpful during
installation to monitor system changes caused by software.
11.3.2 VMware vCloud Log Sources for Compliance
Customers should be able to retrieve logs from all areas that are relevant and unique to their organization.
Retrieval should be possible via a programmatic fashion such as an API to allow for automated queries. Log
collection nodes must be added to a cloud environment, as illustrated below.
App App App
Management
Controls
FW, IDS, AV
App
Regulated
Data
App App
App App
Nonregulated Data
FW, IDS, AV FW, IDS, AV FW, IDS, AV
Distributed Network
ESXi
Hypervisor
Log Collection
Cluster
ESXi
ESXi
ESXi
Log Collection Log Collection Log Collection
Storage Fabric
Physical
Storage
Server
Server
Server
Server
Routers
Switches
Firewalls Load Balancers
Figure 18. Log Collection in the Cloud Environment
VMware vCloud
Logs, generated by VMware components, must be maintained by the provider but also must be available to
tenants. Tenants should be able to download in raw format all vCloud Director and VSE logs pertaining to their
organizations/networks. Logs with customer identifiers should be flagged or indexed for retrieval. The following
diagram illustrates architecture of vCloud components and log collection.
Log Reporting Archive
Log Analysis
Log Analysis
vSE
eVDC
VMware vCloud Director
vCenter Orchestrator
vCD
Server
vCD
Server
vCC
vCD DB
vCC DB
vCenter
vCenter
Server
vShield
Manager
vCenter DB
VMware
vSphere
VMware
vSphere
Customer
Env Log
Collector
VM
VM
VM
VM
vCenter
vShield
Manager
vCenter
Server
vCenter DB
VMware
vSphere
VMware
vSphere
Storage Pools
Figure 19. Architecture of vCloud Components and Log Collection
The following table illustrates to which logs a vCloud tenant must have access.
V M war e C o m p o n e n t
Pr ov i d e r Lo g s
T e n a n t Lo g s

vCenter Server (VC)
vSphere Server (ESXi)
Chargeback Manager (CBM)
vCenter Orchestrator (vCO)
vShield Manager (VSM)
vShield Edge (VSE)
Table 14. vCloud Component Logs
Other components also generate logs in the cloud environment that must be maintained by the provider but
direct tenant access is not required.
VMware vCloud
Ot h e r C o m p o n e n t
Pr ov i d e r Lo g s
T e n a n t Lo g s
vCloud Director DB (Oracle)

VMware Virtual Center Database
VMware vCenter Chargeback
Database
MS SQL Server
Linux (vCD)
Windows System Logs (CBM,
vCO, VC Server)
Table 15. Other Component Logs
Logs in the vCloud Datacenter environment can further be categorized into four logical business layers:
1. Cloud Application: Represents the external interface that the enterprise administrators of the cloud are able
to interact with. These administrators are authenticated and authorized at this layer, and have no (direct or
indirect) access to the underlying infrastructure. They interact only with the Business Orchestration Layer.
2. Business Orchestration: Represents both the configuration entities of the cloud, as well as, the governance
policies to control the cloud deployment. vCenter Chargeback:
Service Catalog: Presents the different service levels available and their configuration elements
Service Design: Represents the service level and specific configuration elements along with any policies
defined.
Configuration Management Database (CMDB): Represents the system of record, which may be federated
with enterprise CMDB.
Service Provision: Represents the final configuration specification.
3. Service Orchestration: Represents the provisioning logic for the cloud infrastructure. This layer consists of
an orchestration director system and automation elements for network, storage, security, and server/
compute vCenter Server (VC), VMware vCloud Director (vCloud Director), vCenter Orchestrator (vCO).
4. Infrastructure Layer: Represents the physical and virtual compute, network, storage, hypervisor, security
and management components vSphere Server (ESXi), vShield Manager (VSM), vShield Edge (VSE).
VMware vCloud
Custom/Enterprise
Portal
API
Administrators
Policy
Application Admin
Application
Management
Users
Cloud Application Layer

Service
Catalog
Service
Design
Enterprise
CMDB
Service
Provision
Business Admin
Business
Management
Business Orchestration Layer
Storage
Automation
Load
Balancer
Storage
AppVM
AppVM
Security
Automation
Orchestration Admin
IDS
Firewall
Service
Orchestration
Network
Automation
Service
Management
Federated
CMDB
Service Orchestration Layer
Server
Automation
AppVM
Cloud Deployment Tier 1

IDS
Firewall
Network
Security
Storage
Compute
Management
Load
Balancer
Hypervisor
Infrastructure Admin
Infrastructure
Management
Infrastructure Layer
Storage
AppVM
AppVM
Physical
AppVM
Cloud Deployment Tier 2
Figure 20. Infrastructure Layers
The abstraction of these four layers and their security controls helps illustrate audit and compliance
requirements for proper authentication and segregation.
Cloud provider administrator accounts, for example, should be maintained in a central repository integrated with
two-factor authentication. Different tiers of cloud deployments (VPDCs) would be made available to enterprise
users. The following diagram illustrates architecture of vCloud components and log collection.
11.4 vCloud Director Diagnostic and Audit Logs

VMware vCloud Director includes two types of logs:
Audit logs that are maintained in the database, and optionally, in a syslog server
Diagnostic logs that are maintained in each cells log directory
The VMware vCloud Director system audit log is maintained in the Oracle database and can be monitored
through the Web UI. Each organization administrator and the system administrator have a view into the log
scoped to their specific area of control. A more comprehensive view of the audit log (and long-term persistence)
is achieved through the use of remote syslog, described below. Log management products are available from a
variety of vendors and open-source projects.
VMware vCloud
Audit events, as defined earlier, are not the only event types. Diagnostic logs, described below, contain
information about system operation events and are stored as files in the local file system of each cells OS.
Diagnostic logs can be useful for problem resolution but are not intended to preserve an trail of system
interactions for audit. Each VMware vCloud Director cell creates several diagnostic log files described in the
Viewing the vCloud Director Logs section of the VMware vCloud Director Administrators Guide.
Audit logs, on the other hand, do record significant actions, including login and logout. A syslog server can be
set up during installation as detailed in the vCloud Director Installation Guide. Exporting the logs to a syslog
server is required for compliance due to multiple reasons:
1. Database logs are not retained after 90 days, while logs transmitted via syslog can be retained as long as
desired.
2. It allows audit logs from all cells to be viewed together in a central location at the same time.
3. It protects the audit logs from loss on the local system due to failure, a lack of disk space, compromise, and
so on.
4. It supports forensics operations in the face of problems like those listed above.
5. It is the method by which many log management and Security Information and Event Management (SIEM)
systems will integrate with vCloud Director. This enables:
a. Correlation of events and activities across vCloud Director, vShield, vSphere, and even the physical
hardware layers of the stack
b. Integration of cloud security operations with the rest of the cloud providers or enterprises security
operations, cutting across physical, virtual, and cloud infrastructures
6. Logging to a remote system, rather than the system the cell is deployed on, provides data integrity,
that is, inhibits tampering. A compromise of the cell does not necessarily enable access to or alteration
of the audit log.
11.5 Load Balancer Considerations

Load balancer considerations are discussed in the following table.
D e ta i l
Security
A front-end firewall is typically deployed before the load balancer. In some

environments additional firewalls may be located between vCloud Director cells
and the resource tiers, including vCenter.
Load Balancers may also provide NAT/SNAT (Source network address
translation) and is typically configured to provide this for the clustered cells.
It is also recommended to secure access between cells and the other
management and resource group components. Refer to the vCloud Director
Installation and Configuration Guide for ports that must be opened.
Single vCloud Director site

and scope
This architecture covers load balancing of a single vCloud Director site or

instance. It does not cover client application load balancing or global load
balancing.
Sizing recommendations
for number of cells
In general, VMware recommends that the number of vCloud Director cell

instances = n + 1, where n is the number of vCenter Server instances providing
compute resources for cloud consumption. Based on the Service Definition, two
vCloud Director cell instances should be sufficient and allow for upgradability
(upgrading one vCloud Director cell, then the other) and high availability.
VMware vCloud
D e ta i l
Requirements for multicell configurations
Multiple vCloud Director cells require NTP (Network Time Protocol) which is a
best practice for all elements of the vCloud infrastructure.
Consult www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf for
more information on how to set up NTP.
Load balancer availability
At least two load balancers in a HA configuration should be used to reduce

single points of failure. There are multiple strategies for this depending on
vendor or software used.
Proxy configuration
Each load-balanced vCloud Director cell requires setting a proxy console IP

address which should be provided by the load balancer in most cases.
Rest API url configuration
The cloud service URL should be mapped to the address provided via the load
balancer. This is configured in the vCloud Director administrator GUI as well as in
the load balancer configuration. This is the address that should be used to check
the health status of the vCloud Director cell.
Awareness of Multi-Cell
Roles
Some vCloud Director cell roles may consume high resources, for example,
image transfer. All cells can perform the same set of tasks but it would be
possible to set policies that effect which ones are used. See the advanced
configuration settings.
Load balancer session

persistence
Sessions are generally provided in secure methods and are terminated at the
cells. Because of this, session persistence should be enabled using SSL.
Load balancing algorithm
Least connections or round-robin is generally acceptable.
vcloud Director cell status

health checks
Each load balancer service should be configured to check the health of the
individual vCloud Director cells. Since each cell responds via HTTPS, this can be
configured quickly via the IP and API end point URL. Load balancers may
support other types of health checks.
Generally services are checked every few-30 seconds based on load. A good
starting point is 5 seconds.
Example GUI URL - https://my.cloud.com/cloud/
Example API URL - https://my.cloud.com/api/versions
In the second example, the versions supported by this end point should be
returned as XML.
Public IP/port
The service IP should be specified appropriately before cells are added to the
service group. Typically port 443 is the only port exposed standard HTTPS.
Advanced configurations
Load balancers can also provide layer 7 content switching or direction, which
may allow a vCloud Director configuration send certain types of client traffic to
dedicated cells. While cells can perform any function, they can be utilized in a
dedicated fashion if they only receive those types of requests.
Connection mapping
When a cell joins an existing cluster, it may try and load balance sessions. This
may impact connection mapping thru the load balancer as it will be unaware of
the balancing happening within the cell cluster.
Session persistence
VMware vShield Edges load balancing functionality does not support SSL
session persistence today and may not be not be suitable at this point for this
application.
Table 16. Load Balancer Considerations
VMware vCloud
12. Appendix: Signed Certificates with vCloud

Director
A certificate is more than just an RSA key for authentication. Additionally, there is a unique serial number for
easy referencing and an identity, which can be a person, computer, or group. The certificate binds the identity to
the RSA key. Certificates also contain a time period for which this binding is considered valid. They can even
contain information on where to verify the certificate. Finally, a purpose or group limitation can be included in
the certificate. An issuer gives out a certificate. To prevent forgery, a cryptographic signature protects the entire
certificate. Certificates enable us to hand out digitally signed identities for hosts and users.
Certificates are written in a specific machine-parsable way called Abstract Syntax Notation One (ASN.1). This is
then wrapped in a specific format such as the Public-Key Cryptography Standards (PKCS) of which there are
various versions.
Because people wanted to be able to use more than just ASCII characters (and because some old network
connections were not even 8-bit clean and could not even safely transport ASCII), these ASN.1 notations are
packed in either the binary DER format, or in the base-64 PEM format. The PEM format is commonly used for
SSL/HTTPS certificates for websites.
Certificates with different purposes are also named differently. To make things more confusing, some vendors
give these certificates or certificate containers different names. The most commonly known X.509 Certificates
are those used in web servers and web browsers. Web servers obtain signed X.509 Certificates from a
commonly known and trusted source. In practice this means from companies like VeriSign / Thawte, but it could
easily be your own Certificates. Another example is the non-profit organization, CAcert.org.
Certificate Type Description
Certificate Authority (CA) - The start of a trust chain. The CA is the only certificate that signs itself.
(Trusted) Root certificate - Another name for a CA certificate. Intermediate Certificate Authorities A CA that is
not signed by itself but by another (parent) CA.
CSR Certificate Signing Request. This is a certificate generated by a person, with an embedded request to sign.
This request is then given to a CA to be signed.
Host (Local Computer) certificate - Certificate issued for a machine (usually signed by a CA).
User (Personal) certificate - Certificate issued for a person (usually signed by a CA).
Private Key - Private key file belonging to a public key of a .pem file. Can be pass-phrase protected.
PKCS#12 (.pl file) - Certificate plus a private key issued and signed by a CA. Normally pass-phrase protected.
PKCS#15 - Certificate standard for Cryptographic Tokens (for instance, USB crypto tokens). Usually PIN
protected. Sometimes multilevel PINs differentiate user and admin access.
CRL - Certificate Revocation List, a list of revoked serial numbers of issued certificates.
St e p
Referenced Document
Generate and import the CA signed wildcard certificate for vCloud Director
Generate the wildcard untrusted certificate with the necessary details
Generate the Certificate Signing Request
Send the CSR to the Certificate Authority
The CA will send you back the cert with a root cert and possibly an intermediate certificate
VMware vCloud
St e p
Referenced Document
import the root certificate into your keystore
Import the intermediate certificate (if there is one, it depends on your CA)
When you run the vCloud configuration script it looks for the correct aliases, you need to create them
by cloning the wildcard alias.
Now import the wildcard certificate for each one
If you need to delete an entry from the keystore
Table 17. Certificate Steps
For further information, refer to the keytool man page keytool(1).

Generating and Importing CA-Signed Wildcard Certificate for vCloud Director
Each vCloud Director host requires two TLSv1/SSL certificates, one for each of its IP addresses. You can use
wildcard certificates to simplify the addition of new Cells. The following is the procedure to create the keystore
for vCloud Director to use. You create the keystore once and then copy it to each new Cell added.
NOTE: The keytool(1) certificate management utility gets installed with vCloud Director, the full path is /opt/
vmware/cloud-director/jre/bin/keytool
Example:
$ mkdir -p /opt/keystore
$ chown vcloud:vcloud /opt/keystore
$ cd /opt/keystore
Generating the Wildcard Untrusted Certificate with the Necessary Details
When you run the following command below, you are going to be asked specific questions in order to generate a
wildcard certificate.
Example:
$ /opt/vmware/cloud-director/jre/bin/keytool -keystore certificates.vmware -storetype JCEKS -storepass
<certificate passwd> -genkey -keyalg RSA -alias wildcard
What is your first and last name?

[Unknown]: johndoe.example.com
What is the name of your organizational unit?
[Unknown]: Cloud Engineering
What is the name of your organization?
[Unknown]: Example, Inc.
What is the name of your City or Locality?
[Unknown]: Cambridge
What is the name of your State or Province?
[Unknown]: Massachusetts
VMware vCloud
What is the two-letter country code for this unit?

[Unknown]: US
Is CN=johndoe.example.com, OU=Cloud Engineering, O=Example, Inc., L=Cambridge, ST=Massachusetts, C=US
correct?
[no]: yes
Enter key password for <wildcard>
(RETURN if same as keystore password):
Generating the Certificate Signing Request
Once the certificate-signing request has been approved, you should be able to obtain the server certificate in
ASCII format. This contains the public key of the certificate that corresponds with the private key that was
created when you generated the certificate request. The ASCII representation of the approved certificate will
look something like:
BEGIN CERTIFICATE
MIICtDCCAl6gAwIBAgIBHTANBgkqhkiG9w0BAQQFADB5MQswCQYDVQQGEwJVUzEO
MAwGA1UECBMFVGV4YXMxDzANBgNVBAcTBkF1c3RpbjEZMBcGA1UEChMQU3VuIE1p
Y3Jvc3lzdGVtczEQMA4GA1UECxMHaVBsYW5ldDEcMBoGA1UEAxMTQ2VydGlmaWNh
dGUgTWFuYWdlcjAeFw0wMTEyMTMyMjQ4MzRaFw0wMjEyMTMyMjQ4MzRaMH0xCzAJ
BgNVBAYTAlVTMQ4wDAYDVQQIEwVUZXhhczEPMA0GA1UEBxMGQXVzdGluMRkwFwYD
VQQKExBTdW4gTWljcm9zeXN0ZW1zMRAwDgYDVQQLEwdpUGxhbmV0MSAwHgYDVQQD
ExdzdW5maXJlLmNlbnRyYWwuc3VuLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAw
gYkCgYEA45ji7uN6LqdCVehxPnuKzzqq2PfFaTaWZhqYro903bSdf9Qp+sGabDfJ
qrspwrgjE2Owwia4H3InHpvzkcf2O2uB89bwm/RyHhU5AGt3wVFmsgN16XIL+smk
CBBSJo31RTuIZw11ZYkkqMZzVY84sBpGJ0mtD1xnWhsb0MYN5bMCAwEAAaOBiDCB
hTARBglghkgBhvhCAQEEBAMCBsAwDgYDVR0PAQH/BAQDAgTwMB0GA1UdDgQWBBTM
q/dM6tawflKUfRnqKupfhU3HlDAfBgNVHSMEGDAWgBRCOcKaQjn6l7Ft1OqsPcji
gwlFuTAgBgNVHREEGTAXgRVuZWlsLmEud2lsc29uQHN1bi5jb20wDQYJKoZIhvcN
AQEEBQADQQApqNPdeDARy6xWu7/SfxAH12S/wPD43OYJqbt/R2y5/Zpde/arIyhk
fucakqo0Bk9DlI/A4IR+b9Q56k6Ce8tO
END CERTIFICATE
Example:
$ /opt/vmware/cloud-director/jre/bin/keytool -keystore certificates.vmware -storetype JCEKS -storepass
<certificate passwd> -certreq -alias wildcard -file wildcard.csr
Send the CSR to the Certificate Authority
The CA will send you back the cert with a root cert and possibly an intermediate certificate.
Import the Root Certificate into your Keystore
Example:
$ /opt/vmware/cloud-director/jre/bin/keytool -storetype JCEKS -storepass <certificate passwd> -keystore
certificates.vmware -import -alias root -file EntrustRootCertificate.cer
VMware vCloud
This step or subsequent certificate import steps may result in a trust this certificate? prompt from keytool(1),
so do not be surprised when this happens. Also be aware that the certificates.vmware) needs to be the same
keystore that the CSR was produced from.
NOTE: The keystore needs to be in JCEKS format, as vCloud Director 1.0 does not support other formats.
Import the Intermediate Certificate
Example:
certificates.vmware -import alias intermediate -file EntrustCrossCertificate.cer
Create Correct Aliases for vCloud Directors Configuration Script
certificates.vmware -keyclone alias wildcard -dest http
certificates.vmware -keyclone alias wildcard -dest consoleproxy
Import the Wildcard Certificate
Example:
certificates.vmware -import -alias http -file wildcard.cer
certificates.vmware -import -alias consoleproxy -file wildcard.cer
Copy the Certificates
It may be obvious to the reader, but careful consideration should be made to ensure that you do not copy the
keystores into the $VCLOUD_HOME location, as this will only confuse you when you run the vCloud Director
configure tool which creates the certificates and proxycertificates files in $VCLOUD_HOME/etc. A more
important note is to ensure that the permissions are set such that the vcloud user has read access, and that the
files are not owned by root with the group and other bits unset).
Deleting an Entry from the Keystore
If you need to delete an entry from the keystore, then you can use the following procedure to do so.
Example:
certificates.vmware -delete alias consoleproxy
Viewing and Verifying the Keystore Entries
The keystore used by vCloud Director is a storage mechanism for the cryptographic tokens. These tokens are
also known as entries, and we will see shortly how to view these keystore entry certificate details using
keytool(1). Each entry in a keystore is identified by a different alias or entry name. Entries also store their last
modified date/time. The KeyStore is also password protected. The password is required to load the keystore
and a password will be requested when saving a keystore for the first time. There are various different types of
KeyStores available, such as:
JKS Java KeyStore. Suns keystore format.
JCEKS Java Cryptography Extension KeyStore.
PKCS #12 Public-Key Cryptography Standards
BKS Bouncy Castle KeyStore.
UBER Bouncy Castle UBER KeyStore.
VMware vCloud
However, currently vCloud Director v1.0 only supports JCEKS, which is a more secure version of JKS.
As previously mentioned, to view the contents of a KeyStore that has already been created, you will need to
know the password, which is a random encrypted string. You can only verify a KeyStore that you have created or
know the password for. The following example shows you how to view the contents of the KeyStore entries.
There are two ways of passing the password to keytool(1). They are as follows:
$ /opt/vmware/cloud-director/jre/bin/keytool -storetype JCEKS -list -v -keystore certificates.vmware
Enter keystore password: <enter keystore password>
and
$ /opt/vmware/cloud-director/jre/bin/keytool -storetype JCEKS -list storepass <keystore passwd> -v -keystore
certificates.vmware
If you do not provide the correct KeyStore password you will see the following error message:
$ /opt/vmware/cloud-director/jre/bin/keytool -storetype JCEKS -list -v -keystore certificates.vmware
Enter keystore password: <incorrect keystore password>
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
java.io.IOException: Keystore was tampered with, or password was incorrect

at com.sun.crypto.provider.JceKeyStore.engineLoad(DashoA13*..)
at java.security.KeyStore.load(KeyStore.java:1185)
at sun.security.tools.KeyTool.doCommands(KeyTool.java:620)
at sun.security.tools.KeyTool.run(KeyTool.java:172)
at sun.security.tools.KeyTool.main(KeyTool.java:166)
$
It is also worth mentioning that the vCloud Director cells do not share a common keystore. As it states in the
installation guide, each cell has its own keystore that contains two keys: one for the HTTP service and one for the
console proxy. If you use a wildcard certificate, you still need two entries in the keystore with the appropriate
aliases and since there is no cell-specific information in the keystore, you could use it for the configure step on
each cell.
It is also important to note that the keystore file should be protected by restrictive operating system
permissions. The password for the file should be stored securely or should be prompted for on server startup.
Things to Check
If you encounter any issues whilst setting up and implementing certificates in vCloud Director, such as the
infamous Cryptographic error then here are a few things to check.
1. Make sure that certificates-vmware is a JCEKS keystore, and not the default JKS type.
2. Make sure that any password for the keystore does not use special characters that the shell will interpret,
when the keystore is created, or that the arguments to keytool(1) were quoted to prevent the shell from
interpreting it (namely the dollar sign followed by anything or asterisk, and so on). There was a case
previously where a customer used something along the lines of -storepass ca$$ and wondered why the
configure script complained that that the password was wrong. It worked from one particular shell because
bash would expand $$ to be the PID of the current shell. Obviously the vCloud Director configure script
does not expand the value and kept using ca$$ instead of ca<someint>
3. Can you successfully list the certificates in the keystore or print it using /opt/vmware/cloud-director/jre/
bin/keytool? Given that the same Java platform code will be used under the covers to read in the keystore,
decrypt values, and so on, this is a good check to make sure that you have not done something wrong.
VMware vCloud
Detailed Example and Output of the vCloud Director HTTP Certificate

$ /opt/vmware/cloud-director/jre/bin/keytool -storetype JCEKS -list storepass <keystore passwd> -v -keystore
certificates.vmware
Keystore type: JCEKS

Keystore provider: SunJCE
Your keystore contains 1 entry
Alias name: http

Creation date: Jan 18, 2011
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
Owner: CN=*.vcloud.vmware.com.us, OU=VMware Architects, O=VMware Inc, L=Hillview Avenue, C=US
Issuer: CN=Entrust Certification Authority - L1C, OU=(c) 2009 Entrust, Inc., OU=www.entrust.net/rpa is
incorporated by reference, O=Entrust, Inc., C=US
Serial number: 3f865g44
Valid from: Tue Jan 18 13:11:31 EST 2011 until: Sat Jan 19 16:28:25 EST 2013
Certificate fingerprints:

MD5: 80:04:E8:70:0E:F1:E8:8B:68:A7:7B:16:C7:69:60:FF
SHA1: B4:C8:82:88:63:2C:E6:08:6C:23:7D:5C:53:4A:C0:54:16:0A:08:88
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.15 Criticality=false

KeyUsage [
DigitalSignature
Key_Encipherment
]
VMware vCloud

SubjectKeyIdentifier [
KeyIdentifier [
0000: 18 F6 DA D8 68 E0 A5 F7 E3 CA 81 C8 56 3C 42 47 ....h.......V<BG
0010: 8F E0 DB F7 ....
]
]
#3: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false

AuthorityInfoAccess [
[
accessMethod: 1.3.6.1.5.5.7.48.1
accessLocation: URIName: http://ocsp.entrust.net]
]

CRLDistributionPoints [
[DistributionPoint:
[URIName: http://crl.entrust.net/level1c.crl]
]]

CertificatePolicies [
[CertificatePolicyId: [1.2.840.113533.7.75.2]
[PolicyQualifierInfo: [
qualifierID: 1.3.6.1.5.5.7.2.1
qualifier: 0000: 16 1A 68 74 74 70 3A 2F 2F 77 77 77 2E 65 6E 74 ..http://www.ent
0010: 72 75 73 74 2E 6E 65 74 2F 72 70 61 rust.net/rpa
]] ]
]
VMware vCloud

ExtendedKeyUsages [
serverAuth
]

BasicConstraints:[
CA:false
PathLen: undefined
]

AuthorityKeyIdentifier [
KeyIdentifier [
0000: 1E F1 AB 89 06 F8 49 0F 01 33 77 EE 14 7A EE 19 ......I..3w..z..
0010: 7C 93 28 4D ..(M
]
]
Certificate[2]:
Owner: CN=Entrust Certification Authority - L1C, OU=(c) 2009 Entrust, Inc., OU=www.entrust.net/rpa is
Issuer: CN=Entrust.net Certification Authority (2048), OU=(c) 1999 Entrust.net Limited, OU=www.entrust.net/
CPS_2048 incorp. by ref. (limits liab.), O=Entrust.net
Serial number: 7245h1sx
Valid from: Fri Dec 11 07:43:54 EST 2009 until: Wed Dec 11 08:13:54 EST 2019

MD5: 2F:B3:00:F2:FA:12:7B:BD:82:95:70:05:96:17:DB:BE
SHA1: 61:43:AF:68:F7:B3:3A:47:94:04:74:98:8B:05:F7:B1:62:96:98:42
Version: 3
VMware vCloud
Extensions:
#1: ObjectId: 2.5.29.15 Criticality=true

KeyUsage [
Key_CertSign
Crl_Sign
]

BasicConstraints:[
CA:true
PathLen:2147483647
]

KeyIdentifier [
0000: 1E F1 AB 89 06 F8 49 0F 01 33 77 EE 14 7A EE 19 ......I..3w..z..
0010: 7C 93 28 4D ..(M
]
]

[
accessMethod: 1.3.6.1.5.5.7.48.1
]

[DistributionPoint:
[URIName: http://crl.entrust.net/2048ca.crl]
]]
VMware vCloud

[CertificatePolicyId: [2.5.29.32.0]
qualifierID: 1.3.6.1.5.5.7.2.1
0010: 72 75 73 74 2E 6E 65 74 2F 72 70 61 rust.net/rpa
]] ]
]

KeyIdentifier [
0000: 55 E4 81 D1 11 80 BE D8 89 B9 08 A3 31 F9 A1 24 U...........1..$
0010: 09 16 B9 70 ...p
]
]
Certificate[3]:
Owner: CN=Entrust.net Certification Authority (2048), OU=(c) 1999 Entrust.net Limited, OU=www.entrust.net/
Serial number: 8392k3du
Valid from: Sat Dec 25 04:50:51 EST 1999 until: Wed Jul 25 00:15:12 EST 2029

MD5: EE:29:31:BC:32:7E:9A:E6:E8:B5:F7:51:B4:34:71:90
SHA1: 50:30:06:09:1D:97:D4:F5:AE:39:F7:CB:E7:92:7D:7D:65:2D:34:31
Version: 3
VMware vCloud
Extensions:

KeyUsage [
Key_CertSign
Crl_Sign
]

BasicConstraints:[
CA:true
PathLen:2147483647
]

KeyIdentifier [
0000: 55 E4 81 D1 11 80 BE D8 89 B9 08 A3 31 F9 A1 24 U...........1..$
0010: 09 16 B9 70 ...p
]
]
*******************************************
*******************************************
Detailed Example and Output of the vCloud Director CONSOLEPROXY Certificate

$ /opt/vmware/cloud-director/jre/bin/keytool -storetype JCEKS -list storepass <keystore passwd> -v
-keystore consoleproxy.vmware
Keystore type: JCEKS

Keystore provider: SunJCE
Your keystore contains 1 entry
VMware vCloud
Alias name: consoleproxy

Creation date: Jan 18, 2011
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
Owner: CN=*.vcloud.vmware.com.us, OU=VMware Architects, O=VMware Inc, L=Hillview Avenue, C=US
Issuer: CN=Entrust Certification Authority - L1C, OU=(c) 2009 Entrust, Inc., OU=www.entrust.net/rpa is
Serial number: 3f865g44
Valid from: Tue Jan 18 13:11:31 EST 2011 until: Sat Jan 19 16:28:25 EST 2013

MD5: 80:04:E8:70:0E:F1:E8:8B:68:A7:7B:16:C7:69:60:FF
SHA1: B4:C8:82:88:63:2C:E6:08:6C:23:7D:5C:53:4A:C0:54:16:0A:08:88
Version: 3
Extensions:

KeyUsage [
DigitalSignature
Key_Encipherment
]

KeyIdentifier [
0000: 18 F6 DA D8 68 E0 A5 F7 E3 CA 81 C8 56 3C 42 47 ....h.......V<BG
0010: 8F E0 DB F7 ....
]
]
VMware vCloud

[
accessMethod: 1.3.6.1.5.5.7.48.1
]

[DistributionPoint:
[URIName: http://crl.entrust.net/level1c.crl]
]]

[CertificatePolicyId: [1.2.840.113533.7.75.2]
qualifierID: 1.3.6.1.5.5.7.2.1
0010: 72 75 73 74 2E 6E 65 74 2F 72 70 61 rust.net/rpa
]] ]
]

ExtendedKeyUsages [
serverAuth
]

BasicConstraints:[
CA:false
PathLen: undefined
]
VMware vCloud

KeyIdentifier [
0000: 1E F1 AB 89 06 F8 49 0F 01 33 77 EE 14 7A EE 19 ......I..3w..z..
0010: 7C 93 28 4D ..(M
]
]
Certificate[2]:
Owner: CN=Entrust Certification Authority - L1C, OU=(c) 2009 Entrust, Inc., OU=www.entrust.net/rpa is
Serial number: 7245h1sx
Valid from: Fri Dec 11 07:43:54 EST 2009 until: Wed Dec 11 08:13:54 EST 2019

MD5: 2F:B3:00:F2:FA:12:7B:BD:82:95:70:05:96:17:DB:BE
SHA1: 61:43:AF:68:F7:B3:3A:47:94:04:74:98:8B:05:F7:B1:62:96:98:42
Version: 3
Extensions:

KeyUsage [
Key_CertSign
Crl_Sign
]

BasicConstraints:[
CA:true
PathLen:2147483647
]
VMware vCloud

KeyIdentifier [
0000: 1E F1 AB 89 06 F8 49 0F 01 33 77 EE 14 7A EE 19 ......I..3w..z..
0010: 7C 93 28 4D ..(M
]
]

[
accessMethod: 1.3.6.1.5.5.7.48.1
]

[DistributionPoint:
[URIName: http://crl.entrust.net/2048ca.crl]
]]

[CertificatePolicyId: [2.5.29.32.0]
qualifierID: 1.3.6.1.5.5.7.2.1
0010: 72 75 73 74 2E 6E 65 74 2F 72 70 61 rust.net/rpa
]] ]
]
VMware vCloud

KeyIdentifier [
0000: 55 E4 81 D1 11 80 BE D8 89 B9 08 A3 31 F9 A1 24 U...........1..$
0010: 09 16 B9 70 ...p
]
]
Certificate[3]:
Owner: CN=Entrust.net Certification Authority (2048), OU=(c) 1999 Entrust.net Limited, OU=www.entrust.net/
Serial number: 8392k3du
Valid from: Sat Dec 25 04:50:51 EST 1999 until: Wed Jul 25 00:15:12 EST 2029

MD5: EE:29:31:BC:32:7E:9A:E6:E8:B5:F7:51:B4:34:71:90
SHA1: 50:30:06:09:1D:97:D4:F5:AE:39:F7:CB:E7:92:7D:7D:65:2D:34:31
Version: 3
Extensions:

KeyUsage [
Key_CertSign
Crl_Sign
]

BasicConstraints:[
CA:true
PathLen:2147483647
]
VMware vCloud

KeyIdentifier [
0000: 55 E4 81 D1 11 80 BE D8 89 B9 08 A3 31 F9 A1 24 U...........1..$
0010: 09 16 B9 70 ...p
]
]
VMware vCloud
13. Appendix: Capacity Planning

Capacity forecasting provides an efficient way to acquire the appropriate amount of physical resources to
support the increased demand for the vCloud. This allows for the growth of the vCloud to be planned and
included in the service providers budgetary process, and reduces the likelihood of panic buying, which
generally increases costs dramatically, and undermines standardization efforts. It also reduces the likelihood of
last minute surprises, such as a lack of available space or power to support the new vCloud infrastructure
components.
From a vCloud perspective, capacity management is both simplified by the existence of the provider virtual
datacenter and organization virtual datacenter constructs, as well as potentially more complicated by the
addition of three models of consumption: pay-as-you-go, allocation (committed), and reservation (dedicated).
Finally, all of these capacity management aspects, within a vCloud context, must address both the cloud (service
provider) administrator as well as the end-customer (organization) administrator perspectives.
Sizing for the workload resource group clusters can be difficult to predict since the provider is not in charge of
what the consumer may run. The provider is also not aware of existing usage statistics for virtual machines that
are run in the cloud. The information below should assist in initial sizing of the vCloud environment and is based
on information from the Service Definition. This information is being provided as examples. It is highly
recommended that you engage you local VMware representative for detailed sizing of your environment.
13.1 Cloud Administrator (Service Provider) Perspective

The primary capacity management concerns of the cloud administrator are:
Capacity management of provider virtual datacenters and the service offerings backed by each
Network capacity management (network bandwidth capacity management is beyond the scope of this
section)
Capacity forecasting
Capacity monitoring and establishing triggers
VMwares vCloud solution makes extensive use of reservations. As such, previous approaches to capacity
management used in vSphere are not as applicable to a vCloud as one might think. CPU and memory overcommitment, for example, cannot be applied as extensively as it once was in a multi-tenant environment.
Capacity management in a vCloud focuses on the vSphere hosts and associated shared storage that comprise
the virtual datacenters.
Unlike managing capacity for vSphere, in a vCloud, the virtual machine is no longer the basis for resource
consumption from a service provider perspective. In the case of a vCloud, the organization virtual datacenter
becomes the basis for resource consumption.
Capacity management is further impacted by the introduction of multiple consumption models in the vCloud
model. Each model requires its own capacity management approach. As a result, this section will provide
guidance for capacity management, from a service provider cloud administrator perspective, as it applies to
each of the consumption models: Pay-As-You-Go, Allocation, and Reservation.
Regardless of the particular consumption model being applied in a provider virtual datacenter, the common
starting point of vCloud capacity management is to calculate the total amount of CPU and memory resources
available for consumption. Since the underlying infrastructure provisioning unit of a provider virtual datacenter is
a vSphere host, the first step is to determine the total CPU and memory at the vSphere host level. The following
table shows the key vSphere host variables needed to calculate capacity, along with example values.
VMware vCloud
It e m
Var i ab l e
Va lu e
Units
Processor Sockets
Nsocket,1
integer
integer
2.4
GHz
64
GB
Processor Cores
Processor Speed
Host Memory
Ncores,1
Sproc,1
Mhost,1
Table 18. vSphere Host Variables
Calculating the total memory available is very straightforward; it is simply the total amount of RAM for the
vSphere host. Total CPU resources are calculated using the following formula:
Phost = NsocketNcores Sproc

Using the example values from the table, the total CPU resources equals 19.2 GHz.
Once the vSphere host capacity model has been defined, the next step is to determine the provider virtual
datacenter (vSphere cluster) capacity. Determining the provider virtual datacenter capacity is critical as vCloud
capacity management should be performed at the provider virtual datacenter level, not the vSphere host level.
When considering vCloud provider virtual datacenter capacity, an additional step is required to ensure that
redundancy has been accounted for. The provider virtual datacenter cluster redundancy may vary depending
upon service levels offered. For the example below, we will assume N+2 cluster redundancy. This means that the
provider virtual datacenter can absorb up to two vSphere host failures and continue to support all hosted virtual
machines at the same level of performance. To accomplish this, there must be capacity available on the
remaining vSphere hosts to take over all workloads.
Based on a requirement for provider virtual datacenter cluster redundancy, the overall number of memory and
CPU consumption units for the provider virtual datacenter (cluster) must be reduced. To determine the
redundancy overhead, the number of vSphere hosts in the cluster and the desired number of redundant vSphere
hosts need to be considered. This is described in the following table:
R e d u n da n c y Var i ab l e s
Nnodes
This represents the number of nodes in a cluster
Nredundant
Rredundancy,HA
This represents the minimum number of redundant nodes

This represents a targeted ratio of redundancy as indicated by a real number
greater than one. This ratio (such as 1.10) indicates that there is a ten percent
overhead committed to availability. For example, a 10 node provider vDC
with a 1.10 redundancy ratio would require 11 nodes to deliver the appropriate
capacity. Note that this level of redundancy may vary depending on the class
of service offering being delivered on that provider vDC.
Redundancy variables can be determined with the equation below.
Table 19. Determining Redundancy Overhead
VMware vCloud
Calculating Redundancy Ratio from Minimal Level of Redundancy
Nnodes + Nredundant
Nnodes
= Rredundancy
For example, the level of redundancy is calculated below for a cluster size of ten nodes containing two
redundant nodes.
Nnodes + Nredundant
Nnodes
)( )
=
8+2
8
= 1.25 = Rredundancy
Once the ratio of redundancy is calculated, the number of units of consumption per provider vDC can be
determined with the following equation:
CPU resources per Cluster
NCPU, cluster Nhosts, cluster PCPU, host

=
Rredundancy, HA
For our example where:
PCPU, host
19.2GHz
This results in:
NCPU, cluster =
8x19.2
1.25
122.88GHz
The number of memory units of consumption is calculated below.

For our example where:
Nmem, host
64GB
This results in:
N
M
Nmem, cluster = hosts, cluster mem, host
1.25
8 x 64
= 409.6GB
1.25
So weve now established that our example provider virtual datacenter has 122.88GHz of available CPU and
409.6GB of available memory, taking a vSphere cluster redundancy of N+2 into account. Next well look at some
guidance for capacity management as it applies to each of the consumption models.
VMware vCloud
Pay-As-You-Go Model
When an organization virtual datacenter is created in the Pay-As-You-Go model, a resource pool is instantiated
with expandable reservations. As such, the customer organization virtual datacenters contained on that provider
virtual datacenter can grow to consume all of the available provider virtual datacenter resources. While this
could be true in any vSphere environment, the added challenge in a vCloud is the use of reservations at the
vApp level. When an organization virtual datacenter is created out of a provider virtual datacenter using the PayAs-You-Go consumption model, a %guarantee is configured for CPU and memory. This is applied to each vApp
or virtual machine within a vApp. For example, if the service provider configures the organization virtual
datacenter with a 50% guarantee for CPU and 75% guarantee for memory, then the customer creates a virtual
machine consuming 1 vCPU of 1GHz and 1GB of memory, a reservation for that virtual machine will be set at 50%
of 1GHz, or 0.5 GHZ and 75% of 1GB, or 0.75GB of memory.
Since there is no way of knowing how a customer will define their virtual machine templates in their private
customer catalogs, coupled with the fact that organization virtual datacenters can expand on demand, VMware
recommends the following:
Calculate the total available CPU and memory resources (less an amount reserved for global catalog
templates), adjusted by the cluster redundancy ratio, at the provider virtual datacenter level
Establish a CPU and Memory %RESERVED threshold at the provider virtual datacenter level
Establish the %RESERVED for the provider virtual datacenter at a number in the 60% range initially
As the total amount of reserved CPU or reserved memory approaches the %RESERVED threshold, do not
deploy new organization virtual datacenters in that provider virtual datacenter without adding additional
resources. If the corresponding vSphere cluster has reached its maximum point of expansion, a new provider
virtual datacenter should be deployed and any new organization virtual datacenter s should be assigned to the
new provider virtual datacenter. In this way there is 40% of expansion capacity for the existing organization
virtual datacenters in the case where the provider virtual datacenter has reached its maximum point of
expansion.
CPU and memory over-commitment can be applied, and if so the %RESERVED value should be set lower than
if no over-commitment is applied due to the unpredictability of the virtual machine sizes being deployed (and
hence reservations being established)
Monitor the %RESERVED on a regular basis and adjust the value according to historical usage as well as project
demand
Allocation Model
When an organization virtual datacenter is created in the Allocation Model, a non-expandable resource pool is
instantiated with a %guaranteed value for CPU and memory that was specified. Using a %guaranteed value of
75%, this means if an organization virtual datacenter is created specifying 100GHz of CPU and 100GB of
memory, a resource pool is created for that organization virtual datacenter with a reservation of 75GHz and limit
of 100GHz for CPU and a reservation of 75GB with a limit of 100GB for memory. The additional 25%, in this
example, is not guaranteed and can be accessed only if its available across the provider virtual datacenter. In
other words, the 25% can be over-committed by the provider at the provider virtual datacenter level and
therefore may not be available depending on how ALL of the organization virtual datacenters in that provider
virtual datacenter are using it.
At the virtual machine level, when a virtual machine is deployed, it is instantiated with no CPU reservation but
with a memory reservation equal to the virtual machines memory allocation multiplied by the %guaranteed.
Despite the fact that no CPU reservation is set at the virtual machine level, the total amount of CPU allocated
across all virtual machines in that organization virtual datacenter is still subject to the overall CPU reservation of
the organization virtual datacenter established by the %guarantee value.
VMware vCloud
Based on this use of reservations in the Allocation Model, VMware recommends the following:
Determine how much resource, at the provider virtual datacenter level, you want to make available for
expanding organization virtual datacenters that are deployed to that provider virtual datacenter
Establish a CPU and Memory %RESERVED (guaranteed, not allocated) threshold at the provider virtual
datacenter level based on the %guaranteed less the amount reserved for growth. The remaining unreserved
resources are available to all organization virtual datacenters for bursting.
virtual datacenter should be deployed and any new organization virtual datacenters should be assigned to the
new provider virtual datacenter. This gives some predetermined amount of capacity available for expanding
the existing organization virtual datacenters in the case where the provider virtual datacenter has reached its
maximum point of expansion.
CPU and memory over-commitment can be applied, but it should be based only on the amount of unreserved
resources at the provider virtual datacenter level, allowing for over-committing the resources available for
organization virtual datacenter bursting.
demand
Reservation Model
When an organization virtual datacenter is created in the Reservation Model, a non-expandable resource pool is
instantiated with the reservation and limit values equivalent to the amount of resources allocated. This means if
an organization virtual datacenter is created allocating 100GHz of CPU and 100GB of memory, a reservation pool
is created for that organization virtual datacenter with a reservation and limit of 100GHz for CPU and a
reservation and limit of 100GB for memory.
At the virtual machine level, when a virtual machine is deployed, it is instantiated with no reservation or limit for
either CPU or memory.
Based on this use of reservations in the Reservation Model, VMware recommends the following:
Determine how much resource, at the provider virtual datacenter level, you want to make available for
expanding organization virtual datacenters that are deployed to that provider virtual datacenter
Establish a CPU and Memory %RESERVED threshold at the provider virtual datacenter level equivalent to the
capacity of the underlying vSphere cluster, taking into account HA redundancy
virtual datacenter should be deployed and any new organization virtual datacenters should be assigned to the
new provider virtual datacenter. In this way there is some predetermined amount of capacity available for
expanding the existing organization virtual datacenters in the case where the provider virtual datacenter has
reached its maximum point of expansion.
No over-commitment can be applied to the provider virtual datacenter in the Reservation Model, due to the
reservation being at the resource pool level
demand
VMware vCloud
Storage
VMware vCloud Director uses a largest available capacity algorithm for deploying virtual machines to
datastores. Storage capacity must be managed on both an individual datastore basis as well as in the aggregate
for a provider virtual datacenter.
In addition to considering VMware storage allocation best practices, manage capacity at the datastore level
using the largest virtual machine storage configuration, in terms of units of consumption, offered in the service
catalog when determining the amount of spare capacity to reserve. For example, if using 1 TB datastores (100
storage units of consumption based on a 10 GB unit of consumption) and the largest virtual machine storage
configuration is 6 storage units of consumption (60 GB), then applying the VMware best practice of
approximately 80% datastore utilization would imply managing to 82 storage units of consumption. This would
result in 82% datastore utilization and reserve capacity equivalent to 3 of the largest virtual machines offered in
the service catalog in terms of storage.
13.2 Network Capacity Planning

A vCloud also brings network capacity planning to the forefront. Providers must consider IP address, VLAN, and
ephemeral port capacity. The following table describes what needs to be managed from a capacity perspective
and its impact:
It e m to M a n ag e
Im pact
IP Addresses
Available IP addresses to be assigned in support of a dedicated external

network for an organization, such as for Internet access or hardwarebased firewall rules
Need to track IP addresses assigned to specific organizations to
determine whats available for a shared external Organization network
VLANs
VLANs available for VLAN-backed pool assignment, if required
Ephemeral Ports
Additional vCloud Director Network Isolation networks that can be

assigned from the vNetwork Distributed Switch (only 1016 ephemeral ports
per vNetwork Distributed Switch), if used
Table 20. Network Capacity Planning Items
VMware vCloud
14. Appendix: Capacity Management

14.1 vCloud-Specific Capacity Forecasting (Demand Management)
Capacity forecasting consists of determining how many organization virtual datacenters are expected to be
provisioned during a specific time period. Capacity provisioning is concerned with determining when vCloud
Infrastructure components must be purchased in order to maintain capacity. From a financial budget
perspective, the procurement of the vCloud Infrastructure requires more planning and understanding of
customer future requirements.
VMware recommends performing two forecasting functions over time.
Capacity Trending. Using historical organization virtual datacenter capacity and utilization data, it is possible to
predict future capacity requirements.
Demand Pipeline. Understanding future customer requirements via the sales pipeline provides the necessary
information to understand future capacity requirements, as well as knowledge of marketing/business
development functions bringing new service offerings to market.
Initially, no historical utilization metrics will be available, and thus it will not be possible to perform capacity
trending for some period of time. During this initial period, a good understanding of the customer demand
pipeline will need to be established. Over time, this pipeline can be combined with trending analysis to more
accurately predict capacity requirements.
The customer demand pipeline must be established in conjunction with the service providers sales teams,
or lines of business (LOB) if a private cloud, so future vCloud capacity requirements can be determined.
This demand pipeline must contain information of all known new customers, expansion of existing customer
organization virtual datacenters, projected sizing metrics, plus any new service offerings that are in
development. The forecasting plan must fit both the budgetary cycle as well as the procurement and
provisioning timeframes. For example, if a quarterly budgetary cycle exists, and the procurement and
provisioning timeframe is one month, it will be necessary to have a pipeline of at least four months to ensure
all requests in the pipeline can be fulfilled.
Over time, capacity trending can be used to assist with the forecasting of organization virtual datacenter
provisioning needs. It uses historical information to determine trends and validates the organization virtual
datacenterforecast based on demand pipeline data.
14.2 Capacity Monitoring and Establishing Triggers

Certain metrics should be monitored carefully to warn of approaching or exceeding consumption thresholds,
and are listed in the following table. These metrics should be measured against each vCloud provider virtual
datacenter and for each organization virtual datacenter within the provider virtual datacenter. To monitor for
threshold breaches, and possible subsequent violation of service level commitments to the cloud consumer,
the appropriate tools and triggers are needed for proper notification.
Attr i b u t e
M o n i to r e d p e r
%RESERVED CPU
provider vDC, organization vDC

Note: For the Pay-As-You-Go consumption model this will be the aggregation of reservations
values for the contained virtual machines.
%RESERVED Memory
CPU utilization
Memory utilization
Datastore utilization
provider vDC
Transfer store utilization
vCloud
Network IP addresses available
vCloud
VMware vCloud
Attr i b u t e
M o n i to r e d p e r
Network IP addresses consumed
organization
Network VLANs available
vCloud
Network ephemeral ports consumed
vNetwork Distributed Switch
Table 21. Capacity Monitoring Metrics
Once thresholds have been exceeded, the group responsible for capacity management of the vCloud should be
notified to add additional capacity. You should account for the time required to add the physical components
necessary to increase the capacity of a provider virtual datacenter. A vCloud-aware capacity management tool
should be deployed. Whichever tool is chosen, the capacity model can be used to forecast new provider virtual
datacenter capacity utilization as well as ongoing capacity management of existing provider virtual datacenters.
It should also account for expansion triggers based on provisioning timeframes.
Once a provider virtual datacenter has had its total amount of available resources calculated, no adjustments to
that provider virtual datacenter such as adding or removing hosts should be made without updating the
calculated value. This model may be altered if long-term CPU and memory reservations are not at the levels that
they were designed for. An increase in the resources allocated to an organization virtual datacenter can affect
the remaining capacity of a full provider virtual datacenter. Monitoring full provider virtual datacenters
should be done on a weekly basis. The resource consumption of virtual machines within an organization virtual
datacenter should be reviewed for trends that indicate the resources purchased for that organization virtual
datacenter are insufficient.
vCenter CapacitIQ, while not vCloud Director aware, can be used to provide insight into provider virtual
datacenter utilization and trends.
14.3 Capacity Management Manual ProcessesProvider Virtual Datacenter

The following cloud administrator capacity management activities include simple, periodic planning activities
supported by operational day-to-day activities. Periodic continuous improvement activities are critical to
extracting the most value from your vCloud Infrastructure.
Planning activities (initially monthly, then quarterly):
Determining usable capacity by provider virtual datacenter and organization virtual datacenter (taking into
account vSphere overhead)
Reviewing current utilization
Reviewing provisioning timeframes for new provider virtual datacenter components (hosts, network, storage)
Forecasting utilization growth over the coming period (preferably based on actual pipeline validated with
historical trending)
Planning for procurement and implementation of additional capacity over the coming period, including bills of
materials and budgets
Reviewing capacity alert threshold levels and setting alerts for capacity warnings
Operational activities (daily):
Monitoring for alerts
Investigating performance issues to determine whether capacity is the root cause
Initiating and managing the procurement and provisioning of additional provider virtual datacenter capacity
Continuous improvement activities (quarterly/yearly):
Comparing capacity model utilization levels to observed levels and tuning model to drive greater utilization
without sacrificing reliability
Optimizing provisioning timeframes (shortening them and making them more predictable)
VMware vCloud
14.4 End-Customer (Organization) Administrator Perspective

The primary capacity management concern of the organization administrator is capacity management of the
organizations organization virtual datacenters.
VMware recommends that all organizations establish a capacity management process based on a standard unit
of consumption. The recommended base Unit of Consumption for each resource important to capacity
management from an organization administrator perspective is shown below.
Attr i b u t e
Var i ab l e
Va lu e
vCPU
PUC
1 GHz
MUC
Memory
1 GB
DUC
Storage
10 GB
Table 22. Organization Virtual Datacenter Units of Consumption
Taking such an approach enables more efficient capacity management since the vApp component virtual
machine resource allocations are predefined in the Service Catalog resulting in vCloud Infrastructure resource
consumption being more accurately predicted.
Each organization will be provided with a finite quantity of resources (in the cases of the allocation and
reservation consumption models) from one or more provider virtual datacenters in the form of organization
virtual datacenters. This means that as the organization consumes the organization virtual datacenter resources,
a tripping point will need to be defined to make sure steps are taken to expand the organization virtual datacenter.
First, the resource consumption limits for an organizations organization virtual datacenters need to be defined,
with these limits defining when action needs to be taken to remove the potential capacity issue.
Attr i b u t e
Var i ab l e
Limit
organization virtual
datacenter CPU
Peak Utilization
CCPULimit
80%
The limit for allocating CPU resources within the

organization virtual datacenter before expansion is
required. This value will vary depending on the
consumption model being used. From an
organization virtual datacenter perspective,
reservation values should be considered equal to
the amount of CPU allocated as reservation values
are not available to the organization administrator.
datacenter Memory
Allocation Limit
CmemLimit
80%
The limit for allocating memory resources within the

organization virtual datacenter before expansion is
required. This value will vary depending on the
consumption model being used. From an
organization virtual datacenter perspective,
reservation values should be considered equal to
the amount of memory allocated as reservation
values are not available to the organization
administrator.
Table 23. Recommended Organization Virtual Datacenter Capacity Thresholds
VMware vCloud
The amount of CPU and Memory resources will vary dependant on the size of the organization virtual datacenter
contracted for. The table below provides an example of the resources needed to calculate the organization
virtual datacenters capacity.
It e m
Var i ab l e
Va lu e
Units
Total organization
virtual datacenter
vCPU Units of
Consumption
SorgvDC
50
GHz
datacenter Memory
Allocation in Units of
Consumption
MorgvDC
64
GB
Table 24. Sample Organization Virtual Datacenter Resource Allocation
The number of capacity units available within this organization virtual datacenter is found by using the following
equations.
Determining organization virtual datacenter memory units of consumption
MUC, orgVDC =
CmemLimit MorgVDC
MUC
Based on the information from the above tables, the total memory unit of consumption for the organization
virtual datacenter is calculated as shown below.
MUC, orgVDC =
CmemLimit MorgVDC
MUC
)(
=
0.8 x 64
1
51.2GB
This results in 51.2 memory units of consumption for the sample organization virtual datacenter.
Determining organization virtual datacenter CPU units of consumption
PUC, orgVDC =
SorgVDC CCPULimit
PUC
Based on the information from the above tables, the CPU units of consumption per organization virtual
datacenter are calculated as shown below.
PUC, orgVDC =

SorgVDC CCPULimit
PUC
)(
=
50 x 0.8
1
40GHz
This results in 40 CPU units of consumption for this sample organization virtual datacenter.
VMware vCloud
14.5 Organization Virtual DatacenterSpecific Capacity Forecasting

Capacity forecasting consists of determining how many virtual machines are expected to be deployed during a
specific time period of the organizations choosing. The time period used for the virtual machine forecast should
correspond to the budgetary process. Capacity provisioning is concerned with determining when an
organization virtual datacenter must be expanded in order to maintain capacity.
VMware recommends that organizations perform two forecasting functions over time.
Capacity Trending Using historical virtual machine capacity and utilization data, it is possible to predict future
capacity requirements.
Capacity Pipeline Understanding future end-user virtual machine(s) resource requirements, via IT and LOB
projects, provides the necessary information to understand future capacity requirements
Over time capacity trending can be used to assist with the forecasting of virtual machine provisioning needs. It
uses historical information to determine trends and validates the virtual machine forecast based on pipeline
data.
Capacity provisioning depends on determining the point of expansion for the organization virtual datacenter.
This is based on determining a point of resource consumption at which the process of procuring and expanding
the organization virtual datacenter must begin so that reserve capacity is not exhausted before the additional
capacity is available. In the vCloud context, this can be considered to be dependent upon the time it takes to
process the purchase request for additional organization virtual datacenter resources. Provisioning time can be
assumed to be zero but is dependent upon specific contractual agreements with the service provider.
The recommended steps to perform capacity trending and to determine a point of organization virtual
datacenter expansion follow.
Regularly Collect Organization Virtual Datacenter Consumption Information
The primary issue with the trending of organization virtual datacenter consumption is identifying the point of
record for all new virtual machines. This can then be used to determine the capacity trends and therefore
determine the overall need for purchasing additional organization virtual datacenter capacity. To establish the
point of record for new virtual machines, the items listed in the following table should be tracked, ideally in a
Configuration Management or Capacity Planning Database as virtual machine attributes.
Var i ab l e
Nam e
Units
orgvDC
Organization virtual
datacenter
This is the organization

virtual datacenter in
which the virtual
machine resides
Identifier
Dbuild
Build Date
This is the date the

virtual machine is built.
Date
NUC,cpu
CPU Units of
Consumption
This is the number of

CPU units of
consumption allocated
to the virtual machine.
CPU Units of
Consumption
NUC,mem
Memory Units of
Consumption
This is the number of

memory units of
consumption allocated
Memory Units of
Consumption
NVGB
Storage
This is the amount of

storage (GB) allocated
GB
Table 25. Organization Virtual Datacenter Trending Information
VMware vCloud
Determine Trending Variables

With the information recorded as described above, it is possible to determine the rate of organization virtual
datacenter consumption.
Var i ab l e
Nam e
Units
Time
This is the time

between points of
observation.
Weeks
NcpuUC
New CPU Units
This is the total number

of CPU units of
consumption required
for the forecasted
virtual machines.
CPU Units of
Consumption
NmemUC
New Memory Units
This is the total number

of memory units of
consumption required
for the forecasted
virtual machines.
Memory Units of
Consumption
NVGB
New Storage (GB)
This is the total amount

of storage required for
the forecasted virtual
machines.
GB
Tpurchase
Organization Virtual
Datacenter Expansion
Purchase Time
The amount of time to

procure additional
datacenter resources.
Weeks
Table 26. Organization Virtual Datacenter Capacity Trending Variables
Determine Standard Rate of Consumption

Determining the Trended Growth Rate
NcpuUC
NmemUC
NVGB
NcpuUC
NmemUC
T
NVGB
T
VMware vCloud
Determining the Trend

It is important to understand that the rate of increase dictates how far in advance additional organization virtual
datacenter resources need to be purchased. The following table presents a sample virtual machine forecast for a
quarter along with sample time to purchase value.
Attr i b u t e
Va lu e
NcpuUC
12
NmemUC
12
NVGB
360 GB
Tpurchase
2 weeks
NcpuUC,cluster
320
NmemUC,cluster
717
Table 27. Sample Organization Virtual Datacenter Trending Information
In this example, NcpuUC,free and NmemUC,free represents the number of free resources within an organization
virtual datacenter at which point additional organization virtual datacenter resources should be ordered. In order
to determine the trigger point for ordering equation 6 should be used if no pipeline data exists.
Determining Trigger Point For Ordering Capacity Using Trends
NUC, free= NCU x Tpurchase

For example, from the data provided below, one would calculate the needed free consumption units as listed in
the following equation, or 24 units.
NcpuUC, free= NcpuUC x Tpurchase = 12 x 2 = 24GHz

NmemUC, free= NmemUC x Tpurchase = 12 x 2 = 24GB
For storage, in this example, the trigger point is calculated at 720 GB:
NVGB, free= NVGB
Tpurchase
360 x 2 = 720GB
Determine the Automatic Point of Expansion

Based on the example above, additional organization virtual datacenter resources would need to be ordered
when the available units of CPU or Memory fall to 24GHz or 24GB respectively, or when storage capacity falls to
720 GB. The additional capacity needs to be on order when described or the capacity will not be available in
time to meet demand.
Currently there are no tools available to assist in organization virtual datacenter capacity management. However,
it it is possible to develop scripts to gather pertinent information with languages such as PowerCLI.
VMware vCloud
14.6 Capacity Management Manual ProcessesOrganization Virtual Datacenter

The following organization administrator capacity management activities include simple, periodic planning
activities supported by operational day-to-day activities. Periodic continuous improvement activities are critical
to extracting the most value from your vCloud.
Planning activities (initially monthly, then quarterly):
Determining usable capacity by organization virtual datacenter
Reviewing current utilization (and performance, where possible)
Reviewing purchasing timeframes for expanding an organization virtual datacenter
Forecasting utilization growth over the coming period (preferably based on actual pipeline validated by
historical trending)
Reviewing capacity alert threshold levels and setting alerts for capacity warnings
Operational activities (daily):
Monitoring for alerts
Investigating performance issues to determine whether capacity is the root cause
Initiating and managing the procurement and provisioning of additional capacity
Continuous improvement activities (quarterly/yearly):
Comparing capacity model utilization levels to observed levels and tuning model to drive greater utilization
without sacrificing reliability.
VMware, Inc. 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www.vmware.com
Copyright 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed
at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be
trademarks of their respective companies. Item No: VMW_11Q1_WP_ArchitectingvCloud_p100_R2

3 - Architecting A Vcloud 1.6 - Technical White Paper PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

3 - Architecting A Vcloud 1.6 - Technical White Paper PDF

Uploaded by

Copyright:

Available Formats

VMware vCloud

4. Providing Cloud Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5.5 vCloud Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

13. Appendix: Capacity Planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

1. What is a VMware vCloud?

Requirements for a Cloud

vCloud Service Definitions

Service Definition for Public Cloud

Service Provider Public vCloud Implementation Example

vCloud Director Installation and Configuration Guide

vCloud API Specification

vSphere Datacenter Administration Guide

vShield Administration Guide

vCenter Chargeback Users Guide

vCenter Orchestrator (vCO)

vCenter Orchestrator Developers Guide

vCloud Request Manager

vCloud Request Manager Installation and Configuration Guide

Table 1. Reference Documentation

1.2 Cloud Computing and vCloud Introduction

1.3 vCloud Components

vCloud Request Manager

Figure 1. vCloud Overview

VMware vCloud Director (vCD)

Cloud Coordinator and UI. Abstracts vSphere resources.

API used to programmatically interact with a vCloud

Underlying foundation of virtualized resources.

Provides network security services

VMware vCenter Chargeback

Optional component that provides resource metering and reporting to

VMware vCenter Orchestrator

Optional component that facilitates orchestration at the vCloud API

VMware vCloud Request Manager

Optional component that provides provisioning request and approval

VMware vCloud Connector

Optional component to facilitate transfer of a powered-off vApp in

Table 2. vCloud Components

VMware vCloud Director (vCD)

vCloud Director Cell

vCloud Director End Users

vShield Manager and vShield Edge Virtual Appliances

Figure 2. Core vCloud Logical Architecture

1.4 vCloud Infrastructure

Reasons for organizing and separating vSphere resources include:

vCloud Consumer Resources

vCloud Director Cells & vCloud Director

vCloud infrastructure virtual machine

vCenter Servers & vCenter Database

vCenter Chargeback Servers

Space allocated to user workloads

Figure 4. vCloud Logical Architecture

2. vCloud Director Constructs

Provider vDC: Silver

Provider vDC: Bronze

Figure 5. vCloud Director Construct to vSphere Mapping

Provider Virtual Datacenter

Logical grouping of vSphere compute resources

A unit of administration that represents a logical

Organization Virtual Datacenter

Subset allocation of a provider virtual datacenters

vApp Templates and Media Catalogs

A collection of available services for consumption.

A network that connects to the outside using an