Deploying Exchange Server 2007 on VMware Infrastructure: A VMware Internal Case Study

. . . . . . . . . . . . 5 Planning and requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 exchange Server 2003: The Legacy environment and Its Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Migration Drivers for Virtualized exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2 . . . . . . . . . . 6 Design & Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .VMware white paper Table of Contents executive Summary. . . . . . . . . . . . . . . . . . . .

analysts and media contacts. The Exchange environment at VMware is deployed on 10 physical servers for all Exchange Server 2007 roles (approximately 40 virtual machines) in a multisite configuration. design considerations and planning approach that were taken by the VMware IT staff. placement. provisioning. internal mail clients. 3) Make the most efficient possible use of their virtual infrastructure by deploying virtual machines at greater densities with equal or better performance through optimal utilization of server. Email and instant messaging are mission-critical services that provide communications and messaging-based notifications for thousands of VMware employees. storage and network infrastructure. non-virtualized architecture. The Microsoft Exchangebased messaging systems at VMware support basic email and integrate with critical workflow tools. detailing the technical design and implementation process. All other roles were deployed on virtual machines. VMware DRS and VMware vMotion™ capabilities of VMware Infrastructure. including 1. the Exchange environment at VMware was based on a traditional. server-side throughput and availability of the Exchange environment are critical requirements that have been met and exceeded by the deployment. Two highly available failover clusters using the Microsoft Cluster Service (MSCS) were used to run all mailbox and public folder servers. By virtualizing their Exchange Server 2007 deployment. Before this project. and company-wide help desk functions. Enhanced end user experience.500 total mailboxes deployed in a single forest/ domain/site AD topology. All Exchange Servers and Active Directory infrastructure servers were deployed in a single-site model. 5) Accelerate Exchange life cycle and management functions with virtual machines and templates for rapid testing. Details of the Exchange Server 2003 environment are as follows: Network Infrastructure • 1 Gigabit per second (1Gbps) switched network between all clustered nodes. release.500 user mailboxes with a “Heavy Average User Load Profile. The virtualized Exchange Server 2007 deployment at VMware supports more than 4 million messages per day. Email is a missioncritical medium for timely communication with key external communities including customers. 2) Execute immediate recovery from host. mailbox servers and other role servers while using robust virtual machine capabilities (design. The benefits realized from virtualized deployment of Exchange Server 2007 are a direct reflection of the VMware IT department’s “Virtualize First” policy that has resulted in more than 85 percent of VMware business applications running in virtual machines today. shared document repositories (public folders) for internal user communities.VMware white paper executive Summary VMware is an industry leading technology company. eliminating single points of failure across various resource/host/application/site components. VMware has gained the ability to: 1) Effectively eliminate planned downtime and maintenance/ upgrade windows on every Exchange 2007 Server using the VMware HA. Scaling the system and making optimal use of existing and new hardware technologies also proved challenging. bug tracking). along with virtual snapshots and clones to help with rapid troubleshooting and problem reproduction. A total of 14 physical servers were required for mailbox roles supporting approximately 7. exchange Server 2003: The Legacy environment and Its Limitations The legacy messaging environment at VMware consisted of Exchange Server 2003 Enterprise Edition in a traditional nonvirtualized deployment. streamlining tasks such as adding user accounts. and reflects the experience of many VMware customers worldwide who are also running Exchange on VMware Infrastructure.5 million external messages (spam included) and supports over 7. 4) Achieve exceptional scalability in a “building block” design. application and site failures through a combination of VMware HA and new clustering technologies (such as clustered continuous replication). load balancing) to enable highly flexible Exchange services. such as engineering notifications (build. and highlighting important operational policies for backups and availability. and backup infrastructure • 10 Megabit per second (10Mbps) switched connectivity for cluster heartbeat communication via unmanaged network switch 3 . This white paper defines the high-level business requirements. In comparison. enable high availability and resiliency against various failure modes. making it costly and difficult to maintain performance levels. the traditional non-virtualized deployment of Exchange Server 2003 required 14 physical servers at a single site for just the mailbox server roles. partners. and manage downtime. This project has been an unqualified success for VMware.” The deployment allows good headroom for growth in a highly available and disaster resilient architecture. The company’s ultimate goal is to virtualize 100 percent of its business applications on VMware Infrastructure to realize similar benefits.

domain and site configuration (all Active Directory servers were virtualized) Exchange Server 2003 Topology • Front-End Servers • Two (2) virtualized Exchange 2003 servers for multiple protocols (IMAP. The original Exchange Server 2003 environment consisted of 14 physical servers which could not adequately support 7.VMware white paper Active Directory Infrastructure • The Exchange 2003 organization was deployed in a single forest. • Heavy IMAP usage. Adding more hardware was not cost-effective. Legacy exchange Server 2003 environment 4 . due to expense for acquisition.app (email client for Mac OS X and iPhone) • Evolution (configured as IMAP client) The legacy Exchange Server 2003 native-OS deployment posed significant limitations: 1.500 mailboxes on existing infrastructure Client Systems and Protocols in use • MAPI • Outlook 2000/2003/2007 clients • WebDAV/HTTP protocol • Outlook Web Access (OWA) • ActiveSync • Entourage 2004/2008 (Microsoft e-mail client for MacOS) • Evolution (E-Mail client for Linux) • Miscellaneous IMAP clients • Thunderbird (IMAP email client by Mozilla) • Mail. Scalability of the environment was a serious issue.500 heavy user mailboxes. 2 passive) Microsoft Cluster with four (4) mailbox servers for new hire employees of VMware • Total of 7. and a desire to reduce datacenter footprint. POP3 and HTTP) • Hardware load balancer provided even distribution between clients and the front-end servers • Back-End Servers • Eight node (6 active/2 passive) Microsoft Cluster o Five (5) mailbox servers for all of North and South America users o One (1) mailbox server serves all Asia Pacific/EMEA users • Six-node (4 active. The user population and email requirements at VMware had grown rapidly during the life of the system. Figure 1. • Heavy I/O and large mailboxes were extremely common. management and operations. since content-conversion took place on the mailbox servers.

Uneven distribution of mailbox loads required frequent mailbox movement between the mailbox servers to ensure proper load balancing and consistent end-user experience. This leads to stability issues that included memory fragmentation. 5. Migration Drivers for Virtualized exchange The limitations inherent in the legacy Exchange 2003 deployment were significant reasons to take the opportunity to redesign and redeploy the messaging infrastructure in virtual machines. 6. it is unable to use modern large memory and cache effectively. snapshots and clones can help with very rapid troubleshooting and problem reproduction. 4. Testing in a virtual environment enabled very rapid changes. 7. VMware concluded that combining the benefits of Exchange Server 2007 benefits with the benefits of virtualization benefits created a winning combination. VMware took this opportunity to ensure that running Exchange 2007 on VMware Infrastructure would offer very significant additional 5 . Easier Testing and Prototyping It was very important to accelerate the life cycle of design/test/ stage/production and not be curtailed by the limitations of traditional non-virtualized deployments. 4) Highly scalable infrastructure – Adding users and mailbox servers/roles based on VMware Infrastructure is reduced to a simple building blocks approach based on robust virtual machine design/placement/load balancing/scaling models. Production backup schedules were difficult to meet. availability. small and inefficient use of allocated database cache and kernel exhaustion. 3. 2) Immediate recovery from host. improvements in performance. resiliency. 5) Expedite Exchange Management tasks and accelerate life cycle – virtual machines and templates provide rapid testing. Physical server configurations are more difficult to provision and maintain. While evaluating the performance and 64-bit architectural enhancements in Exchange Server 2007 from an upgrade perspective. Early in the deployment of Exchange. Traditional non-virtualized deployments precluded consolidation of the mailbox server instances to leverage larger and more powerful servers. and had a greater negative impact on the production systems during normal operation. 9. Reboots and system outages were required to complete simple server hardware upgrades. and effective testing on less hardware. These benefits include: 1) Dealing much more effectively with planned downtimes and maintenance/upgrades by using technologies such as VMware vMotion and VMware DRS. testing the design under simulated load was essential to ensuring that Exchange 2007 would operate as expected. A single point of failure exists in Single Copy Clusters (SCC) deployed with SAN storage. but imposed potentially significant down times that were not acceptable as an only option. provisioning. Storage. the only remedy is a reboot of the Exchange server and downtime for its users. and an extended outage. As the initial design process was completed. cluster or site failure would require recovery of the data from tape. The main drivers toward the virtualized Exchange Server 2007 deployment at VMware are listed below: VMware “Virtualize First” Policy VMware follows a “Virtualize First” policy when evaluating application deployments and upgrades so as to realize important benefits enabled by VMware Infrastructure. lose of connectivity etc are a significant vulnerability. 3) Better use of hardware resources – VMware Infrastructure enables greater density of virtual machines that can optimally utilize various server/storage/network resource.VMware white paper 2. Corruption of the information stores or SAN infrastructure inconsistencies such as improper zoning procedures. In this situation. Availability levels were reduced overall because of the planned application and hardware upgrades. load balancing/ scalability and stability issues. 8. As it became obvious that Exchange Server 2007 in and of itself could alleviate some scalability issues. mailbox quotas were not enforced. Tape-based disaster recovery was adequate as a last resort. This design improved agility for IT Operations by leveraging VMware Infrastructure capabilities such as vMotion and VMware HA. Exchange Server 2003 and its 32-bit architecture is limited to 4GB effective memory. life cycle management while reducing overall costs. Backups took longer as users were added. lack of mailbox quotas in the Exchange Server 2003 environment led to unpredictable server / storage utilization and mailbox growth. As the company’s growth accelerated. Provisioning new hardware to replace older Exchange Servers is extremely difficult. in addition to evaluating a transition to Exchange Server 2007. scalability. application and site failures – combination of using VMware HA capabilities and traditional clustering technologies (MSCS) – to eliminate single points of failure across various resource/host/application/site levels.

4. administration. VMware vMotion makes this a simple exercise. and backups/restore. and enabling better scale up on new and much more powerful 64-bit hardware were also important considerations behind the virtualized deployment. the needs for planned hardware upgrades and related performance issues. testing and correcting as necessary. but would increase expense for hardware. there are no “off hours” at VMware. power and cooling. Additionally. Create a migration plan based on Business and Technical requirements using a pilot-to-beta-to-production approach. availability. 3. 9. Establish a baseline from the existing Exchange 2003 deployment to measure its capabilities and limitations across dimensions such as availability. VMware ensured that the planning phase resulted in a solid design framework that would meet or exceed clearly defined business requirements. Using more. scalability. The combination of VMware HA and the application failover capabilities built into Exchange Server 2007 CCR can effectively eliminate single points of failure inherent in a nonvirtualized implementation. storage sizing and network topology. along with the VMware “Heavy Average User Profile. Clearly define the overall workload for the Exchange 2007 deployment. Build the production environment based on the final validated test environment. 7. 6. Someone will always notice a mail outage. and can significantly reduce outages resulting from single-server issues. Collect and rationalize commercial and operational requirements from across the company. Detailed planning started in January 2008. Site Resiliency VMware required a rapid response to any failure ranging from a single resource to loss of an entire site. 11. JetStress. Translate and map the business requirements for scaling. Transition existing Exchange 2003 mailboxes to virtualized Exchange Server 2007 production environment. and other parameters into a set of architectural and design plans. With the cooperation of the various teams. Analyze and remediate any observed anomalies. and factor in key assumptions for scaling. migrations began March 2008 and the entire environment was fully virtualized and complete by June 30.VMware white paper Scalability and Server Containment The demands of the messaging infrastructure at VMware and heavy mail usage user profile required more compute resources. 10.” resulted in an unstable environment because of application downtime caused by memory fragmentation. 5. Size the infrastructure components. Virtualization creates an opportunity to reduce server footprint and preserve the option to use more powerful servers later. including servers. Enhanced Availability Maintenance tasks are a fact of life for production servers. VMware IT staff enlisted assistance from various groups within VMware. upgrades or replacements without interruption to the service. With all Exchange servers in virtual machines. 2008. 6 Planning and requirements Key Planning Considerations and Steps VMware IT Operations teams began a proof of concept of Exchange 2007 on a fully-virtualized platform in October 2007. As a worldwide company. and assure a rapid return of services in scenarios less dire than a true regional catastrophe. Product Engineering and the VMware Business Solutions unit. adjust the design as necessary and iterate through the tests. Validate the test environment using Exchange Server 2007 testing utilities (LoadGen. growth and headroom. 8. Develop a test environment comparable to the proposed architecture. including Performance Monitoring. every server can be moved while running to allow hardware maintenance. 2. disaster recovery. The legacy environment relied on a single-site design with a tape-based recovery process. datacenter space. VMware DRS will automatically move a virtual machine to an appropriate destination when a server becomes heavily loaded. These capabilities require virtualization. including VMware ESX hosts. ExBPA). Monitor new production environment for anomalies. Leveraging 64-bit hardware capabilities fully through Exchange Server 2007. it still cannot be designed in accordance with Microsoft best practices in a non-virtualized deployment that utilizes the entire capability of a modern server. . they also require server downtime. storage and networking. recoverability. The approach VMware took in the planning phase was as follows: 1. backups. Quality Assurance. Application Stability The inherent limitations of running Exchange Server 2003 in a traditional IT environment. While Exchange 2007 is able to exploit modern large memory architectures more effectively. performance. virtual machines. The business requirement for a much more stable and inherently scalable environment was a major factor behind virtualized deployment. smaller servers would avoid wasting compute resource. Unfortunately.

A brief list of advantages (more exhaustive information can be found at http://www. VMware HA and VMware DRS enable the required up time during both scheduled and unscheduled downtime for host patching and upgrades. reliable. which translates to more optimal use of storage. Leveraging VMware Infrastructure Capabilities The ability to scale Exchange mailbox servers on one physical server was a major driver for virtualization of the Exchange infrastructure. guest backups.microsoft. 4. VMware HA protects all Exchange roles.Provide flexible options around backups (with VMware Consolidated Backup. Ensure that the system is capable of immediate failover of the messaging systems with minimal to zero IT interaction and without significant impact to connected clients. VMware HA was chosen to address the majority of cases where simple hardware failure in a non-clustered server can compromise overall operation. the VMware IT organization can utilize larger physical servers running VMware ESX 3. Continue to support a very “Heavy Average Mail User” profile. as well as peripheral roles (hub transport. Design Considerations Architectural Enhancements in Exchange Server 2007 Exchange Server 2007 is designed to exploit 64-bit architectures. POP. 7. etc). Reduce overall costs associated with operating and growing our user base/messaging needs.com/exchange) include • Ability to use 64-bit memory addresses – scales better with more memory .VMware white paper Business Requirements VMware places very high demands on its messaging systems and requires that they be available 24x7x365. resilient and mobile virtualized messaging infrastructure that is capable of withstanding site failure. client access. VMware HA is simple to set up and protects every virtual machine without requiring complex clustering software. 2. improving overall availability when compared to a physical deployment. 9. 6. Support for multiple protocols – A significant percentage of the VMware mail users connect to the Exchange environment via IMAP. Ensure the ability to maintain two (2) synchronized copies of the Exchange Server 2007 data. VMware HA minimizes the duration of any outage by automatically restarting all virtual machines on another VMware ESX server in the event of server hardware outage. 32GB maximum recommended configuration. VMware had the following enhanced business requirements as it planned the migration over to the new Exchange Server 2007 messaging environment: 1. Improved Performance – Reduce latency for connected clients. 3. 5. Deployment of new Exchange 2007 virtual machines from preconfigured templates significantly reduced the time to provision. High Availability – Elimination of any single point of failure in the architecture. 10. drove the decision to continue use of Microsoft clustering technologies in conjunction with VMware HA. Ensure that the infrastructure enables fast recovery from outages due to server or information store outages or corruption. and HTTP clients.Application Aware Clustering The dependency of VMware on messaging as a mission-critical application. As a result. Ensure that significantly improved I/O performance in Exchange Server 2007 translates to a better end-user experience with faster Outlook response times and reduced RPC latency. Zero-downtime upgrades – Use VMware vMotion along with Microsoft Clustering services to enable patching and upgrading with minimal impact to end users.5 to host multiple mailbox servers. Very High Resiliency . Quota Implementation – Control mailbox size and growth to allow predictable utilization of storage resources and reduce administrative overhead associated with mailbox movement. VMware HA minimizes any disruption to the messaging environment. While clustering can improve availability for mailbox servers. • Reduced input/output (I/O) requirements enabled by the larger memory caches available on 64-bit systems – also leads to better CPU scaling. array-based snapshots and passive node backups) 8. Scalability – Use VMware Infrastructure to increase hardware utilization efficiency. and resulting high availability requirements. Flexible backup options . • Typically results in much better I/O performance and reduction in I/O requirements. With VMware Infrastructure.mailbox sizes and the number of user accounts per server increase. Stacking multiple virtual machines on powerful physical servers enables better hardware efficiency and utilization while helping to overcome the Microsoft recommended Exchange maximum of eight processor cores and 32GB of RAM. have a much more inherently scalable operational foundation. Multi-Site Resiliency – Provide a robust. and to overcome the Exchange 2007 eight-core. 7 . and can be easily tested to ensure that the environment recovers as expected.

VMware decided on the use of blade technology to reduce the physical space and cooling requirements that were placed on the datacenter. Disk input/output rate (I/O per second. etc.aspx). On the other hand. VMware HA automatically restarts virtual machines on other VMware ESX hosts in a virtual cluster in the event of an ESX host failure. This is based on the 500 heavy-user-per-core benchmark which was achieved both in-house and externally. all other peripheral roles are allocated 4GB of memory each. virtual disks can be used to deliver consistent and complete backups of virtual machines via snapshots. client access servers. if the I/O requirement per GB is high. These building blocks easily satisfy the requirements of the four active mailbox servers and can still accommodate the various peripheral roles that may run on any one physical VMware ESX host at one time. or IOPS) are a key design consideration for Exchange. The building block VMware IT utilizes is based on four virtual CPUs. raw disk mapping – Internal testing confirmed that VMware could meet I/O requirements regardless of whether virtual disks or raw disk mappings were used. If the I/O requirement per GB is low. system performance will be seriously degraded unless the smaller drives are selected or the larger drives are only partially used. VMware has documented (http://www. The choice of more powerful blades allows the infrastructure to scale and reduces overall TCO.vmware. This approach could also be facilitate “swinging” LUNS back to physical servers for troubleshooting.200 GB. This type of trade-off highlights the importance of working closely with the storage vendor of choice to determine optimal disk and I/O capabilities based on performance and space requirements. wasting the extra capacity. Choosing 146 GB drives instead delivers the same IOPS rate for 584 GB. This required more administrative effort to 8 .html ) optimized deployment models and best practices based on a “building block” approach. but reduce the available IOPS per GB of data. which supports VSS. The design utilizes RAID10 for all database and log LUNs. Exchange CCR removed the architectural dependency on shared storage and eliminated exposure to a complete site outage as a single point of failure (a core requirement of the design). or require that those large drives be only partially utilized.com/solutions/ business-critical-apps/exchange/resources.microsoft. and ensures physical separation between the two to ensure recoverability and optimum performance. the lower cost per GB option might be appropriate. Utilize Multicore Capabilities Exchange deployments on physical servers are not able to effectively utilize the compute power now available in more powerful quad-core (and more) systems. Each site has a single EMC CX3-80 SAN Storage array for storage of the virtual machines and all local Exchange data. IOPS demands may effectively prevent the use of fewer high-capacity drives. In some cases. In the legacy VMware messaging environment. several factors were taken into consideration: • Virtual disk vs. Storage Considerations When designing the storage solution for the virtualized Exchange 2007 environment. Raw device mappings in physical compatibility mode was selected to accommodate array based clones or snapshots. Several mailboxes had grown past 10GB in size. A group of 4x300GB 15k rpm drives can deliver approximately 500 – 600 IOPS for 1. user mailbox quotas were not enforced.com/en-us/library/bb124521. VMware ESX helps to resolve this issue and enables significantly more scalability for the virtualized Exchange 2007 environment.5. 64GB of Memory per Blade Server Using 64GB of memory per blade server enables maintaining lower cost by utilizing smaller 4GB memory modules. The VMware IT mailbox server building block utilizes 12GB of memory (2.VMware white paper Exchange 2007 Cluster Continuous Replication (CCR) “is a high availability feature of Microsoft Exchange Server 2007 that combines the asynchronous log shipping and replay technology built into Exchange 2007 with the failover and management features provided by the Cluster service” (http://technet. • Disk layout & quotas –The storage configuration conforms to EMC and Microsoft best practices. Note that with the release of VMware Consolidated Backup 1.000 users at 5MB mailbox cache and 2GB for the operating system). 300GB drives provide high capacity. For example. Exchange 2007 CCR on top of VMware HA allows VMware to have application-aware high availability. virtual disks were slightly faster. • Performance – Disk / spindle count is a significant issue that relates directly to the balance between designing for capacity versus throughput.). Having 16 cores available in a single blade server enables deploying up to four active mailbox servers per blade and still leaves plenty of resources/headroom available to support passive mailbox servers and peripheral roles (hub transport. as well as host-level high availability.

• This results in a highly balanced and optimized Exchange configuration. CCR affords the ability to replicate the Exchange server data within the datacenter known as “in datacenter service” or to a remote site datacenter known as “site resiliency”. Utilizing the VMware Infrastructure Feature Set Optimally To further improve upon the new design. Due to the VMware HA cluster. • Replication can now reverse direction to primary site update to synchronize the Exchange. No mailboxes above 5GB will be supported. The Exchange 2007 environment is 100 percent virtualized on VMware ESX v3. 9 . • Virtual machines will be migrated to the host in the cluster that will satisfy the request using vMotion. Scenario: • Exchange Mailbox. database. a third site was chosen to house the file-share witnesses used by the CCR clusters to maintain quorum. The majority of the user population maintains a mailbox below 2GB. automated failover and multiple synchronized copies of the mailbox data and site resiliency. VMware HA will restart any virtual machine from a failed physical node on a running node in the HA/DRS cluster. 10 physical servers run the entire virtualized Exchange Server 2007 organization including the domain controllers. All of the virtual machines in the environment (including active mailbox servers) can be migrated under load with vMotion. CAS or HUB server or servers require additional compute resources • VMware DRS scans the cluster to determine the host that can best satisfy the request for additional resources. the following capabilities within the VMware Infrastructure feature set were utilized: VMware High Availability (HA) – Allows virtual machines to be restarted on other nodes of a VMware ESX Server cluster in the event of hardware failure. VMware HA is utilized to ensure the virtual machines are powered on in the case of a server hardware failure. Standard quotas of 2GB and 5GB are enforced across all mailboxes. The legacy Exchange Server 2003 environment had 14 mailbox servers alone. The implementation of user mailbox quotas mitigates this additional administrative effort. anti-spam and perimeter e-mail routing servers. To achieve complete autonomy between datacenters.VMware white paper maintain manageable Exchange Server information stores by constantly moving mailboxes. the CCR cluster will automatically move the clustered mailbox role to the passive node in the second datacenter and continue to provide email services. One HA/DRS cluster is deployed in both of the VMware datacenters. this causes the cluster to fail over to the remote site. Exchange Server High Availability To meet the business requirements of no single point of failure. VMware Distributed Resource Scheduling (DRS) – VMware DRS continuously monitors utilization across resource pools and will intelligently allocate resources to virtual machines that require additional resources based on pre-defined policies. and CCR will initiate reverse replication to ensure that the cluster is in sync. The same methodology would be used in the case of a total site failure. Enterprise Edition. Scenario: • Host server fails. VMware vMotion – Allows virtual machines to move from one physical server to another physical server with no impact on operation or end user services.5. To place this in the proper perspective. the failed active clusternode will be restarted on any remaining available ESX host. • VMware HA restarts all Exchange virtual machines on another host in the local cluster. Users with special circumstances or business justifications for an increased mailbox size can obtain 5GB of mailbox storage with the appropriate approvals. VMware DRS is used with full automation level for all Exchange server roles including clustered mailbox servers to allow for auto-placement based on resource requirements. The virtual machines run Windows 2003 x64 R2 SP2. VMware has achieved a consolidation ratio of about four to one (4:1) with approximately 40 virtual machines running on 10 physical servers. In the event of an unscheduled ESX host outage. Design & Implementation With the transition to Exchange Server 2007. • Failback can occur manually when the Exchange Administrator is available. The CCR design used at VMware populates each HA/DRS clusters with eleven CCR cluster-nodes. VMware combines VMware HA with a new Microsoft Exchange Server 2007 applicationlevel high availability replication technology known as Cluster Continuous Replication (CCR). VMware vMotion is used to move virtual machines between physical servers within the VMware HA cluster for physical server maintenance and patching.

• Affected ESX host can be placed in Maintenance mode. all connections to Santa Clara mailboxes via OWA. • Virtual machines migrate to other ESX hosts in the cluster freeing up the affected host. In this design. IMAP and POP connections are also proxied through Palo Alto. Client access and mail routing at VMware 10 . • ESX host is patched and additional memory is added. fewer overall Exchange servers and fewer customizations. In this model. fewer domain controllers. The two Active Directory design also meant changes to how client access worked. However. the majority of infrastructure services are served from two datacenters near the company’s main campus in Palo Alto.one or two Active Directory sites? One Active Directory site would allow for simplicity. which required customization of the default IMAP and POP Exchange Server 2007 Environment Figure 2. and Outlook Anywhere are proxied to Santa Clara through Palo Alto. The two-datacenter design of the Exchange 2007 infrastructure and the nature of mail routing in Exchange 2007 led to an interesting dilemma . California. We chose to use a two Active Directory site configuration. ActiveSync.VMware white paper Scenario: • A VMware ESX host needs to be updated via patches and additional memory must be added to the host. it was possible that mail between two mail users whose mailbox resided on the same mailbox database could potentially bounce between the Palo Alto and Santa Clara datacenters. Two Active Directory sites would allow for all mail local to Palo Alto or Santa Clara to stay within their respective datacenters using local hub transport servers. Mail Routing and Client Access VMware uses a centralized infrastructure and administration model. • ESX host is restarted and virtual machines are migrated back to upgraded host.

a second Active Directory site was created for Santa Clara. client access and domain controller/global catalog servers. and ensures physical separation between the two to ensure recoverability and optimum performance. Each site has dedicated hub transport. The four spindle LUN allows us to satisfy our 1. Figure 3 – Virtual Machine Disk Layout 11 . The addition of this site helps avoid mail from unnecessarily being sent across the metro link. Each site has a single EMC CX3-80 SAN Storage array for storage of the virtual machines and all local Exchange data. The virtual machines are deployed as 22-mailbox cluster nodes that make up 11 Cluster Continuous Replication (CCR) clusters.VMware white paper Exchange Server 2007 Environment The new Exchange Server 2007 environment runs entirely in virtual machines on VMware Infrastructure 3. Storage Subsystem Architecture/Design The storage configuration leverages our storage vendor’s best practices. System/boot partitions use virtual disks on VMware VMFS volumes configured for RAID5 at the array level. The cluster nodes are geographically dispersed between the Palo Alto and Santa Clara datacenters. The standard mailbox server building block uses seven (7) database LUNs configured as physical mode RDMs. The design utilizes RAID10 for all database and log LUNs. Transaction log LUNS are configured using ten (10) 300GB/15k FC spindles. Each database LUN houses two (2) mailbox databases for a maximum of 350 users.15 per mailbox I/O requirement. This design allows for site resiliency and eliminates single point of failure. This houses 11 highly available Exchange Server 2007 clustered mailbox servers (CMS). To efficiently route traffic between the various transport servers. Database LUNs are configured using four (4) 300GB/15k FC spindles. The single log LUN is configured as a physical mode RDM and is shared between the 14 storage groups that reside on the standard mailbox server building block.

Furthermore. Backup environment SG1. 13 SG7.8 SG2. The use of VTLs allows VMware IT to keep 30 days worth of backup data online and quickly accessible. This allows for the implementation of a less stringent backup schedule such as weekly full and daily incremental (or differential). 14 Sun Full incr incr incr incr incr incr Mon incr Full incr incr incr incr incr Tues incr incr Full incr incr incr incr Wed incr incr incr Full incr incr incr Thurs incr incr incr incr Full incr incr Fri incr incr incr incr incr Full incr Sat incr incr incr incr incr incr Full Figure 5.9 SG3.11 SG5. recovery from alternate backup media would only be required in the event of multiple failures. one at the primary (active) data source and one at the alternate (passive) data source.10 SG4. Due to the tape-free design. Figure 4.VMware white paper Performing Backups in the Exchange Server 2007 Environment Daily backups are performed using traditional Exchange streaming backups to virtual tape libraries (VTL). there is no risk of having tapes off-site when needed or having to locate all of the different tapes used for a particular save-set. 12 SG6. Backup Schedule 12 .

VMware white paper Hardware Configuration Figure 6. hp c7000 Blade Chassis Storage Area Network Configuration Figure 7. SaN Configuration 13 .

VMware white paper Figure 8. VMware eSX Servers/exchange infrastructure Virtual Machine Figure 9. exchange Server 2007 Cluster and Site Configuration 14 .

exchange Server 2007 topology Conclusions VMware has successfully deployed a fully virtualized Exchange Server 2007 environment on VMware Infrastructure. By virtualizing its Exchange infrastructure. This project has been a complete success. 15 .” 4) Evaluate and leverage advancements in hardware technologies with minimal/zero disruption on the production Exchange environment. 3) Continually monitor and optimize the physical and virtual infrastructure to continue to drive down costs associated with space. power/cooling and enabling the transition to “green IT. such as acquisitions and rapid user group provisioning/data migrations. The benefits of virtualization combined with the benefits of Exchange 2007 offer capabilities and cost savings that are not otherwise available. This deployment has met and exceeded the business and technical requirements that were defined and which have been outlined in this paper. 2) Deal efficiently with discontinuous points of growth. and reflects the consistently positive experience that VMware customers report after deploying Exchange 2007 on VMware Infrastructure. VMware has gained the ability to: 1) Easily scale up and continuing to grow the deployment as the company’s base of users grows.VMware white paper Figure 10.

102. 7.961.725.vmware. 6.806.496.820. All rights reserved.086. 6.951. 7.156. 7. 7.925.492.155.278. 6. 7.030.847.998. Inc.022.424.397.789.683. in the United States and/or other jurisdictions.149. 7.886.785.815.598.S. 6. 6.966.242.710.961.082.487.277.409.413. VMware is a registered trademark or trademark of VMware.145. 7. 7.221. 7. 7. All other marks and names mentioned herein may be trademarks of their respective companies.136. 7.222. 6.281. 7.880.999.275. 7. 7. 6.412.679. 7.704.428.VMware. 3401 Hillview Ave Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www.735.481.111. 6.111.843.941. 7. 6.253. Protected by one or more U.260.356.com Copyright © 2008 VMware. 6.938.558.601.672.944.636. 7. . Patents Nos.260.711.412.377. 7.289.002 and patents pending. 6. 7.434. 7.069. Inc.277. 7.089.269. 7. Inc. 7. 6. 7.795.117. 7. 6.433.702. 6.