You are on page 1of 21

Virtualizing VMware Zimbra with VMware vSphere and NetApp NFS

Reference Architecture Study

T e c h n i c a l W HI T E P A P E R

Virtualizing Zimbra with VMware vSphere and NetApp NFS

Table of Contents Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Target Audience. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Introduction to VMware Zimbra. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Zimbra High-Level Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 VMware vSphere Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Why use NetApp NFS for Zimbra?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 NFS Tuning on NetApp Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Storage Efficiency with NetApp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 NetApp Storage: Aggregates and RAID Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 High Availability Design with NFS and NetApp Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Solution Building Block. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Storage Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Test Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Test Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Access Type and Activity Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Test Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Data Amounts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Data Integrity Checks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Summary of Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23


Virtualizing Zimbra with VMware vSphere and NetApp NFS

Executive Summary
Businesses are always looking for creative ways to increase worker productivity and lower IT costs. With messaging, businesses cannot afford to deploy sub-standard systems, because this creates unnecessary outages, drastically reduces end-user productivity and increases costs. To accomplish these goals, messaging solutions can include file-based storage designs which can reduce complexity and cost over block-based storage designs. VMware Zimbra Collaboration Server is a next-generation email and collaboration platform that provides a rich end-user experience and allows IT architects, systems administrators and virtual infrastructure administrators to have design and implementation flexibility. At the same time, it reduces the cost of deploying, operating and maintaining an enterprise messaging system when virtualized. This paper will focus on the benefits of virtualizing the Zimbra Collaboration Server using VMware vSphere and the NetApp storage platform, Network File System (NFS). For our reference architecture study, we created and tested a heavy enterprise user configuration that easily scales to 100,000 users. This paper covers the Zimbra, vSphere and NetApp storage configuration for this enterprise study. We will discuss our test results, as well as some of the more important features of virtualization such as simplified disaster recovery, live migration with VMware vMotion and automated load balancing with VMware Distributed Resource Scheduler (DRS).

Target Audience
This paper is intended for: System architects and administrators Email architects and administrators vSphere architects and administrators VMware and NetApp customers Anyone interested or involved in the design and implementation of email and collaboration systems Anyone interested in the migration of an existing or legacy collaboration system to VMware Zimbra The VMware Zimbra Web site, located at, contains information about the best practices for design, implementation and migration of existing or legacy email or collaboration products to Zimbra.

Introduction to VMware Zimbra

Zimbra Collaboration Server is a next-generation collaboration platform that supports email, document sharing and group calendar using an AJAX Web interface and enables drag-and-drop functionality, right-click context menus and tool tips. It is composed entirely of open source software and offers extremely competitive cost savings when compared to other commercial products such as Microsoft Exchange and IBM Domino. The collaboration server is based on the Linux operating system and currently supports most 64-bit distributions of Linux, such as Red Hat Enterprise Linux, Ubuntu and SUSE Linux Enterprise server. Zimbra is also supported on Mac OS X. Zimbra is available in two editions: Open Source and Network. The advantage to deploying the Network Edition over the Open Source edition is full support from VMware for the core product. Support is also provided for all of the additional features included with the Network Edition, such as the Zimbra Connector for Outlook , the Zimbra Connector for BlackBerry Enterprise Server and Zimbra migration tools. Zimbra also includes archiving and discovery, including cross-mailbox search. Additional features include: Email, contacts, calendar and documents Sync to other desktops and devices Unlimited storage


Virtualizing Zimbra with VMware vSphere and NetApp NFS

Zimbra High-Level Architecture

The Zimbra mailbox server consists of following components: A MySQL database holds the metadata including the pointers to blobs. The metadata also holds pointers to user contacts, address book and calendar data, as well as other items. Redo logs that store the records of every transaction on the Zimbra mailbox server. They also provide point-intime backup and restore. In the zmbackup function, the redo logs are the backup. A primary store, commonly referred to as the database or datastore, is responsible for storing all incoming email. It consists of a journaling Linux file system that stores flat files on this system until they are moved to the secondary store for near-line storage using the HSM component of the Zimbra product. In the unlikely event of a mailbox server crash, the journaling file system can be recovered by replaying transactions. One or more secondary stores for the HSM component hold the users mailbox data after it has been moved to the secondary stores based on the configurable policy settings in the HSM component of the Zimbra product. A Lucene index component of the Zimbra mailbox server is responsible for indexing users messages and attachments for instantaneous search capability. These components comprise the core Zimbra mailbox server. The entire product includes the following components. These can be deployed on separate virtual machines in a multi-server environment or installed in a single server environment, such as with the VMware Zimbra Appliance. OpenLDAP (Lightweight Directory Access Protocol) server This is most commonly referred to as the directory. This component holds all of the account data including usernames, passwords, aliases, user mailbox server locations, and the settings of mailbox server and user attributes (such as spam settings). A minimum of two OpenLDAP servers should be deployed for redundancy and load-balancing. Zimbra Message Transfer Agent (MTA) This component is responsible for receiving and delivering user email internally and externally. The Zimbra MTA server is capable of running anti-spam/antivirus, though most enterprises usually have some external appliance for this purpose and in most cases will disable anti-spam/ antivirus on the MTA. Zimbra proxy server Zimbra uses a consolidated webmail, POP and IMAP interfaces located on the Zimbra mailbox server (aka the MBS server). Secure HTTP, IMAP and POP services are accessed on separate proxy servers. It is therefore recommended to expose a virtual IP or VIP URL through the use of the HTTPS proxy. This can provide load balancing and redundancy for webmail connections.


Virtualizing Zimbra with VMware vSphere and NetApp NFS

Figure 1. VMware Zimbra Architecture and its Components


Virtualizing Zimbra with VMware vSphere and NetApp NFS

VMware vSphere Configuration

The reference architecture in this document is based on vSphere 4.0 Update 2, which was later updated to vSphere 4.1. The vSphere cluster was constructed from four IBM x3650 hosts equipped with two Quad Core Nehalem processors and 32GB of system memory. All four IBM x3650 hosts were configured for VMware HA and VMware DRS. The HA cluster is configured with Percentage Reservation Admission Control policy, given that the Zimbra component virtual machines have various resource reservation requirements. The amount of resource reservation is set at a conservative 20 percent. The VMware DRS load balance is set at normal threshold. Once all virtual machines were distributed and operational, anti-affinity rules were created to place virtual machines on hosts so that no host would contain any multiple of the Zimbra messaging application roles. For example, host1 was configured so that it could only contain one Zimbra mailbox server. In Figure 2, we show the anti-affinity rule for the vSphere cluster separating the Zimbra mailbox servers onto separate hosts within the cluster. In the event a host containing a mailbox server should fail, VMware DRS will locate another host within the cluster that does not already contain a Zimbra mailbox server, register and boot that mailbox server on the newly chosen host.

Figure 2. VMware vSphere Anti-Affinity Rutle Configuration

Four network interface adapters (NICs) were installed and configured in each of the hosts. The NICs were configured according to VMware and NetApp best practices (as documented in NetApp TR3749) in the following manner: For the VMkernel Interface for NFS Storage, two VMNICs were teamed with Route base on IP hash load balance algorithm. NOTE With Out-IP load balance algorithm, link aggregation is required on the physical network switches, with LACP disabled Two VMNICs were also teamed for the Service Console, virtual Machine network and vMotion interfaces with default setting of Source port ID load-balancing algorithm.

Virtualizing Zimbra with VMware vSphere and NetApp NFS

NOTE No additional configuration is needed on the physical switch for the default load balance algorithm. We overrode the active/standby NIC order to have the traffic split between both interfaces without sacrificing availability in the event of a NIC failure. In Figure 3, we show the configuration for vmnic1, which has been assigned to the Service Console and vMotion. Note that vmnic1 is active for Service Console and vMotion, and standby for virtual Machine network, while vmnic2 is active for virtual Machine network, and standby for Service Console and vMotion.

Figure 3. vmnic 1 Configuration

Figure 4. vmnic 2 Configuration


Virtualizing Zimbra with VMware vSphere and NetApp NFS

On the vCenter Server that manages the ESX servers, the NetApp Virtual Storage Console 2.0.1 (VSC) plug-in was installed. VSC 2.0.1 is available on NetApps NOW Web site located at as a free downloadable plug-in. This plug-in provides the following functions that ESX needs to perform when using NetApp storage on the backend: Sets the required NFS parameters on the ESX servers that mount the datastores from the NetApp storage with the recommended values with a single click. Provisions the NFS datastore to grow or shrink in size on the fly. Clones virtual machines instantly and has ability to point the clones to a different datastore. Provides tools to align the virtual machines created for running the different Zimbra components. The scripts to set the right Guest Operating System (GOS) time-out values are also available as the part of this plug-in. The following best practice recommendations were observed and followed while setting up the virtual machines before installing Zimbra: Use the vmnet3 network driver while creating the virtual machine. This driver is available when VMtools are installed on the virtual machine. Set the recommended NFS parameters using VSC on the ESX hosts. Align the newly created virtual machine using fdisk or mbralign as documented in NetApp TR-3747. Run the GOS time-out script from VSC after the GOS is installed in the virtual machine that automatically sets the timeout value for the Linux OS. Use VSC to clone the newly created gold virtual machine to the required number of virtual machines for Zimbra. This cloning process leverages NetApp FlexClone technology to create instant, space efficient virtual machine clones. No manual virtual machine deployment was needed, and the cloning of VMs was done almost instantaneously. While creating multiple VMDKs in a single virtual machine, separate SCSI bus IDs are used to spread out the load on different SCSI buses. All datastores were provisioned to the ESX cluster using VSC for efficiency and consistency. Modify the ESX hosts so that all Zimbra virtual machine swap space (VSWP file) is stored on a separate datastore. Using this separate location for the Zimbra transient data improves backup efficiency when using a new datastore as the backup location.

All storage backing the Zimbra components in this reference architecture is NFS based. This approach is different from a traditional NFS implementation in which the Zimbra mailbox servers would typically mount a NFS volume within the GOS. Instead, all NFS volumes are provisioned to ESX hosts and mounted as datastores. By doing this, write order and write-through integrity are both satisfied as I/O now goes through VMware vSphere virtual SCSI emulation. All VMDKs used by Zimbra virtual machines are seen as SCSI disks within the virtual machines guest operation system, which is no different than those that reside on a VMFS volume over block storage protocols such as Fibre Channel and iSCSI. For more information on this subject, refer to VMware KB 1012143.


Virtualizing Zimbra with VMware vSphere and NetApp NFS

In Figure 5, we show a representation of the network and storage connection configuration for NetApp storage used in this reference architecture.

Figure 5. vSphere and NetApp Storage Lab Layout and Configuration


Virtualizing Zimbra with VMware vSphere and NetApp NFS

In Figure 6, we show a representation of the NetApp storage in relation to the ESX hosts and the Zimbra virtual machines. In this screenshot, the NFS datastores are presented to all of the ESX hosts and the Zimbra virtual machines see this storage as SCSI disk drives.

Figure 6. NetApp NFS Storage and Zimbra Virtual Machines Layout and Configuration

Why use NetApp NFS for Zimbra?

VMware vSphere uses the Network File System (NFS), Fibre Channel and iSCSI protocols to provide storage to virtual machines more commonly known as datastores. Setting up Zimbra on virtual machines mounting datastores over NetApp NFS provides good value to the entire architecture. The following points worked to the solutions advantage when running the various components of the Zimbra architecture on datastores mounted over NetApp NFS: Zimbra virtualized using vSphere on NetApp storage proves that mail servers and mailboxes can function and perform effectively over NFS. NetApp NFS eliminates the 2TB datastore limitation, thus providing Zimbra with the datastores required for extremely large mailboxes. NetApp NFS is thin provisioned by default and thus reduces initial storage consumption. Policy-driven growing and shrinking of a datastore on demand is possible for VMDK files over NetApp NFS. Using VCS 2.0 role-based access enables controls to the storage system hosting the Zimbra datastores to be set. Virtualizing Zimbra on NetApp NFS is simple, easy to manage and troubleshoot. A larger number of virtual machines can be accommodated with a fewer number of datastores. The vMotion, HA and DRS features in vSphere are all supported over NetApp NFS and proved to work perfectly in the Zimbra architecture.


Virtualizing Zimbra with VMware vSphere and NetApp NFS

Zimbra virtualized on vSphere over NetApp NFS can scale both from an I/O performance as well as a capacity perspective. Data deduplication savings is realized at the volume level; datastore size can shrink dynamically with data deduplication.

NFS Tuning on NetApp Storage

Earlier in the tests, the NFS receive window size was set to 26280 by default. What this means is that a NFS client cannot send one 32k write request in one TCP stream and will have to send two TCP packets to complete one write request. Since VMware ESX uses 64k read/write sizes, we can increase the window size so that the NFS stack can receive all that data together in one write request. The nfs.tcp.recv windowsize controls the TCP receive window size for NFS. The earlier default value of 26280 was increased to 65940, and we now recommend this new default value for improved networking performance. These values are normally used as the defaults values and are changed as needed. We changed these values to improve Zimbra platform performance: options nfs.tcp.recv windowsize 65940 options nfs.tcp.xfersize 65536

Even after these values were set on the storage, performance results obtained after running our tests indicated that there was a NFS flow control state in the stack that led to a higher latency within the Zimbra mailbox servers. To mitigate the latency, the following options were set on the storage array to mitigate the NFS flow control state and the latency dropped to the required level for the Zimbra mailbox servers to provide an excellent end-user experience. optionsnfs.sendq.max controls the maximum number of asynchronous messages that can be held up by one client at one time. What this means is that once there are that many requests being processed by the filer from one client, if that client sends any more requests, the subsequent messages will have to wait for asynchronous messages to become available before any more requests get processed. This allows requests from other clients to be processed at the same time and does not allow one client to tie up all NFS resources on the filer. We set this value to 128. options nfs.sendq.max 128

NFS TCP flow control options nfs.ifc.rcv.high and nfs.ifc.rcv.low represent the flow control high water mark and flow control low water mark. The server goes into flow control once the TCP packet data crosses the high water mark. The server comes out of flow control once the TCP packet data comes down below the low water mark. In our testing, we set these options to the following values: options nfs.ifc.xmt.high 64 options nfs.ifc.xmt.low 48

The flow control high water mark adjusts automatically when the NFS TCP receive window size is changed.

Storage Efficiency with NetApp

NetApp offers a variety of storage efficiency technologies that can reduce the amount of storage required by the VMware environment. As with data protection, these features work with all protocols but provide the greatest benefit in NFS environments. Deduplication A VMware datastore has an inherently high level of data duplication, because it typically contains many copies of the same guest operating systems, application software, etc. NetApp deduplication technology can eliminate this duplicate data, dramatically reducing the amount of storage required by the VMware environment. Space savings in NFS datastores can range from 50-90 percent with 70 percent being typical.


Virtualizing Zimbra with VMware vSphere and NetApp NFS

Thin provisioning Thin provisioning allows oversubscription of a VMware datastore, allocating more total space to the virtual machines using a datastore than actually exists. This is based on the assumption that few virtual machines will use all the space allotted to them. Thin provisioned VMDK files are the default option with NetApp NFS to reduce initial storage consumption. Thin provisioning can be used with both NFS volumes and LUNs. However for LUNs, there is typically no mechanism for the underlying file system (VMFS in this case) to communicate to the storage system when space has been permanently freed such as when a VMDK file gets deleted. Space reclamation processes are needed to get back this unused space. With NFS volumes, the storage system can immediately recover the freed space on thinly provisioned volumes without requiring any action from the storage administrator. This also provides for easier storage management. Cloning In almost any VMware environment there is a need to copy or clone existing virtual machines for testing, provisioning or other purposes. NetApp FlexClone creates instantaneous clones of existing virtual machines while consuming almost no additional capacity. The NetApp rapid cloning utility (RCU) part of virtual Storage Console (VSC) is a management plug-in for VMware vCenter that helps automate virtual machine level and datastore level cloning, automatically customizes the guest operating system of each cloned virtual machine, and imports cloned virtual machines into VMware vCenter 4.1. VSC also integrates provisioning, resizing, deduplication, and deletion of datastores.

NetApp Storage: Aggregates and RAID Groups

Data ONTAP allows users to configure individual disk drives into large blocks of storage called aggregates. The individual disk drives contained in an aggregate are configured in RAID groups. NetApp recommends using RAID-DP parity protection when creating RAID groups and aggregates. RAID-DP uses two parity disks and can provide protection from the simultaneous loss of two data disks in a RAID group. RAID-DP provides significantly better data protection compared to more traditional single parity RAID schemes (RAID 5), while providing significantly better storage utilization compared to RAID schemes that offer more than single disk parity protection through mirroring (RAID 1). NetApp recommends using the default RAID group size for optimal performance, but for the Zimbra setup a RAID group size of 24 (22+2) was used with three RAID groups for capacity and throughput requirements on the faster fiber channel disks. For our reference architecture study the latency sensitive parts of the Zimbra install, such as /opt/zimbra, /opt/zimbra/redo, /opt/zimbra/store, /opt/zimbra/index and /opt/zimbra/db all reside on fibre channel disks. A RAID group size of 16 (14+2) was used with a single RAID group for the slower SATA disks. The SATA disks are used for storing the /opt/zimbra/store02, /opt/zimbra/backup and /opt/ zimbra/logs. Even though the SATA disks are slower compared to the fibre channel they are higher capacity and better suited for storing less frequently accessed data that has a higher latency tolerance.

High Availability Design with NFS and NetApp Storage

NFS uses a different model for HA design than block-oriented storage. The first difference is that block-oriented protocols (iSCSI, FC) use an initiator-to-target multipathing model based on MPIO. The domain of the path choice is from the initiator to the target. For NFS, the domain of link selection is from one Ethernet MAC to another Ethernet MAC. This is configured from the host to the switch, the switch to the host, and the NFS server to the switch and the switch to the NFS server. The following mechanisms are used to select from one link or another: A link aggregation choice, which is setup per TCP connection and is either static (setup once and permanent for the duration of the TCP session) or dynamic (can be renegotiated while maintaining the TCP connection but still always on one link or another). A TCP/IP routing choice in which an IP address (and the associated link) is selected based on a layer-3 routing choice. Clustering at the storage level provides another level of data availability and resiliency for business continuance. When the storage controllers are in a cluster, the partner instantly takes over all the operations from the failing node thus avoiding any disruptions to the application.


Virtualizing Zimbra with VMware vSphere and NetApp NFS

It is recommended that HA and DRS are configured at the ESX hosts and the NetApp storage controllers for high availability of data. In Figure 7, we show two NetApp FAS6080 filers configured in a highly available clustered configuration.

Figure 7. NetApp FAS6080 Filers in a NetApp Cluster Configuration

Solution Building Block

In order to ensure the ease of design, deployment and scalability of this messaging solution, we used a concept similar to the process that we used to simplify the design and provisioning of similar solutions on vSphere referred to as the VMware building block. Using this process, we took the best practices for the Zimbra application and combined it with the best practices for CPU, memory and storage and created a building block that recreates predictable performance, scalability and is easily deployed. The building used in our study consists of an enterprise-class 5,000-user profile on a single virtualized Zimbra mailbox server. Similar building blocks were used for other Zimbra components that we will describe later in this section. For our 10,000 heavy enterprise user reference architecture, we needed to deploy the following virtual machines into the configured vSphere cluster. We will detail the configuration of the virtual machines in later sections. We deployed the following machines using the NetApp cloning tool: LDAP Total of two virtual machines; one master one replica MTA Total of two virtual machines for redundancy and load balancing Proxy Total of two for load balancing via virtual IP. These were not utilized MBS Total of two mailbox servers each hosting 5000 heavy enterprise mailboxes Client Total of two SOAP client machines for loadgen testing configuration The major resources that were used for the solution building block in this study are the vSphere v4.1 datacenter product and the NetApp FAS-6080 filer. While the building block used in this study may not be appropriate in


Virtualizing Zimbra with VMware vSphere and NetApp NFS

all enterprise environments, the concept should be helpful as a starting point for most medium to large enterprise environments looking to virtualize or consolidate their physical Zimbra deployments. The following table illustrates the solution building block for the Zimbra virtual mailbox server.
VM Numbers Operating System Hardware (virtual) Processor Memory 2 (SLES) Linux 64-bit v11.0 VMware ESX v4.1 4 vCPU virtual machine on Nehalem based server 16GB allocated to the virtual machine out of 32GB host memory used in reference architecture VMXNet-3 adapter used in reference architecture See Table 3 below Approximately 1700 IOPS An estimated 200Mbps Service Network An estimated 280Mbps Storage Network /opt/zimbra /opt/zimbra/db /opt/zimbra/store /opt/zimbra/redo /opt/zimbra/index /opt/zimbra/store02 (secondary stores) /opt/zimbra/logs

Network Enhanced VMXNet Storage Random IOPS (write 60-70% and includes logging) Throughput/bandwidth

Mount points

Table 1. Zimbra Virtual Mailbox Server Sizing Requirements

In Table 1, we initially estimated that the virtual machine(s) that would host the Zimbra mailbox servers would need to be configured with four virtual CPU and at least 16GB of memory allocated to the virtual machine. We configured one VMXNet-3 adapter per virtual mailbox host to gain the maximum network performance. We also distributed the virtual machines across the vSphere cluster to ensure that we could achieve balanced resource utilization across all ESX hosts in the cluster. Observing Zimbra application requirements and best practices, we configured a mix of NetApp Tier-1 and near-line storage for this study. For the Tier-1 storage, we provisioned fibre channel backed storage and SATA based storage for the backup and hierarchal storage management functions. We calculated the amounts of tier-1 and near-line storage and I/Os per second (IOPs) that would be required for our reference architecture.

Storage Layout
In our calculations, we found that five of the seven storage mounts or partitions required tier-1 storage and three partitions required near-line storage. Using the NetApp FAS 6080 filer we created three aggregates as follows: OS This aggregate hosts all of the operating system disks for the virtual machines Tier-1 This aggregate hosts all of the partitions that require fast performing storage Near-line This aggregate hosts the HSM and logs mounts for the Zimbra application


Virtualizing Zimbra with VMware vSphere and NetApp NFS

After we created appropriate datastores on the aggregates, we began to layout the virtual disk files for the mailbox servers on the created datastores. Listed below are the layouts for the mailbox servers by storage type: Datastore_mbox_fast: Provisioned from 300GB 15k RPM fiber channel disks Mounted as VIF_IP2 Created VMDK files each representing one of the following file systems /opt/zimbra /opt/zimbra/store /opt/zimbra/redo /opt/zimbra/db /opt/zimbra/index Datastore_mbox_slow: Provisioned from 1TB 7.2 RPM SATA disks Mounted as VIF_IP1 Created VMDK files representing each of the following file systems: /opt/Zimbra/store02 /opt/Zimbra/log /opt/zimbra/backup Datastore_OS: Provisioned from the FC aggregate  Mounted as VIF_IP1 For placement of OS virtual disk for the different virtual machines (mailbox, LDAP, proxy, MTA) In Table 2, we provide the configuration for the virtual machines used in this reference study. These include the proxy servers, message transport agents (MTAs) servers, mailbox servers and the LDAP servers. Antivirus and anti-spam were not enabled on the MTAs as we used an external gateway to perform these functions.
Function vC P U Memory OS Disk N e t wor k


2 vCPU 2 vCPU 2 vCPU 2 vCPU 2 vCPU 2 vCPU 4 vCPU 4vCPU

4GB 4GB 4GB 4GB 4GB 4GB 16GB 16GB

32GB 32GB 32GB 32GB 32GB 32GB 32GB 32GB



Virtualizing Zimbra with VMware vSphere and NetApp NFS


vC P U


OS Disk

N e t wor k




32GB 32GB


Table 2. Virtual Machine Configurations

Test Methodology
To validate the integrity of the underlying data necessary for the Zimbra application to function normally within the aforementioned overall environment, a series of tests were performed. The goal of the tests was to introduce an extremely high level of email activity, exceeding that of a typical enterprise-level use pattern. The test were performed in a progressive manner, providing an increased amount of data in the various structures under the mount points where growth is normally seen (/db, /store and /index). Most importantly, the tests also included the introduction of various failure scenarios involving complete loss of connectivity between components of the overall solution during the high email activity test periods. This was followed by various integrity checking methods. Test Tool A Zimbra designed and implemented load generation tool was used for the tests. The tool is strictly used for in-house purposes and is not currently available for public consumption at this time. Written in Java, the Load generator has the following features: A parser to interpret the specific test tasks which can be defined in XML. A task scheduler to schedule the test tasks for different test accounts. All tasks are placed in a queue waiting to be picked up and executed at the scheduled time. Various test task implementations which simulate users interaction with ZCS through different protocols (e.g., SOAP, Html, IMAP, POP3, CalDAV, BES sync, etc.). The load generation tool relies on mapping user actions to SOAP calls that are initiated from a test client through the communication of certain protocols to the Zimbra servers, ultimately simulating user activity. This activity, as it relates to the Zimbra servers through the automated requests produced by the load generator, represents the equivalent of having a simulated number of simultaneously active users. Access Type and Activity Pattern The 10,000 accounts were loaded in the Zimbra environment across two Zimbra mailbox servers (5,000 per mailbox server virtual machine), with a breakdown of email access and activity as follows: 90 percent SOAP users, 10 percent IMAP users receive 21 emails/hour from external compose/send seven emails/hour, four of which go to internal (Zimbra) users 120KB average message size 25 percent of the messages with an attachment of 500KB reads 24/hour moves 3/hour 100 percent Concurrency (above activity pattern used on all 10,000 accounts simultaneously) Test Sequence Before the tests were performed, each of the 10,000 mailboxes was populated with messages to reach a certain size (this is performed through a cloning operation with the Load Generator tool). Once this is done, the load generation

Virtualizing Zimbra with VMware vSphere and NetApp NFS

tool is executed numerous times, to create the activity, such that there is progressive growth in blobs, mySQL, index, and provide an opportunity to check data integrity. As mentioned previously, during some of the load generation tests, failure scenarios were introduced. The following summarizes the sequence of tests performed. Test #1 Execute load generation, initiate NetApp cluster failover Test #2 Execute load generation for 1 hour Test #3 Execute load generation, initiate network disconnect (disconnected both Ethernet cables out of NetApp filer simultaneously) Test #4 Execute load generation, initiate network disconnect (disconnected both Ethernet cables out of NetApp filer simultaneously) Test #5 Execute load generation, power off ESX host (HA kicks in) Test #6 Execute load generation, initiate NetApp cluster failover, let tests complete Test #7 - Execute load generation for 1 hour Test results validated that in a heavy Zimbra activity scenario, when one storage partner of the NetApp cluster failed, the cluster takeover completed in less than nine seconds upon a fault injection and there was no interruption to the users and no data corruption was observed after validating the database with Zimbra integrity checking tools. Other failures such as simulated network outages did not yield any disruptions or data corruption as well. Data Amounts Final data amounts on the larger of the two Zimbra mailbox server virtual machines.
Filesystem Size Us e d Ava i l a b l e % Us e d Mounted on

/dev/sda3 /dev/sda1 /dev/sdb /dev/sdc

38GB 92MB 49GB 147GB

25G 29MB 13GB 460MB

12GB 59MB 36GB 143GB

69% 33% 26% 1%

/ /boot /opt/zimbra /opt/zimbra/ redo /opt/zimbra/ store /opt/zimbra/db /opt/zimbra/ index /opt/zimbra/ store2 /opt/zimbra/log /opt/zimbra/ backup






/dev/sde /dev/sdf

195GB 2.0TB

24GB 142GB

168GB 1.8TB

13% 8%






/dev/sdh /dev/sdi

49GB 2.0TB

626MB 471MB

48GB 1.9TB

2% 1%

Table 3. Zimbra Mailbox Virtual Machine Storage Configuration


Virtualizing Zimbra with VMware vSphere and NetApp NFS

Data Integrity Checks After every one of the seven tests runs, on both mailbox server hosts and the LDAP hosts, we ran the following commands to check for corruption: MySQL Integrity Checks (using native integrity checking utilities and data dump): ./zmdbintegrityreport v (spawns mysqlcheck) . /mysqldump -u root --password=2Ip7pmPV7P2BnA9QzOQswzoTbuLSst --socket=/opt/zimbra/db/ mysql.sock --all-databases --single-transaction>>/tmp/mysql.zimbra Visual inspection of opt/zimbra/log/mysql_error.log LDAP Integrity Checks (searches and data export) zmprov l gaa ldapsearch -LL -x -h zmldap01.zimbralab.local -p 389 -D uid=zimbra,cn=admins,cn=zimbra zimbraId=* -w netapp1 uid | wc -l ldapsearch -LL -x -h zmldap01.zimbralab.local -p 389 -D uid=zimbra,cn=admins,cn=zimbra zimbraId=fb373c97-caaf-47f9-ac69-5e19413cc988 -w netapp1 ldapsearch -LL -x -h zmldap01.zimbralab.local -p 389 -D uid=zimbra,cn=admins,cn=zimbra cn=mbs2* -w netapp1 uid ldapsearch -LL -x -h zmldap01.zimbralab.local -p 389 -D uid=zimbra,cn=admins,cn=zimbra (&(objectclass=zimbraAccount)(!(cn=mbs1*))) -w netapp1 cn ldapsearch -LL -x -h zmldap01.zimbralab.local -p 389 -D uid=zimbra,cn=admins,cn=zimbra (&(objectclass=zimbraAccount)(!(zimbraId=fb373c97-caaf-47f9-ac69-5e19413cc988))) -w netapp1 cn | wc l ./openldap/sbin/slapcat F /opt/zimbra/data/ldap/config b l /tmp/zimbra-ldap.ldif Results were recorded on the SOAPGen client hosts We performed an inspection of client.err and mysql_error.log (client.err is a SOAPGen log only, not a normal Zimbra log) We also performed an inspection of the Zimbra mailbox log (grep for corruption, exception, error, etc.) After the 7th test, zmblobchk on both MBS hosts was run


Virtualizing Zimbra with VMware vSphere and NetApp NFS

Summary of Results
The testing configuration mimicked a typical 10,000 user enterprise email configuration with very heavy profiles and 2GB mailbox sizes. This configuration allowed us to determine how the Zimbra Collaboration Server would perform under extreme enterprise loads. The tests examined messages sent/received, moves and deletes, as well all of the other functions that an enterprise mail user would perform on a regular basis. All of our testing was performed without incident. Unlike Exchange and its LoadGen tool, the databases or LDAP infrastructure were not reset to perform continued testing. As a result of continued testing without resetting the databases, we realized incremental data growth which gave us better insight into how data growth on the system would impact the performance of the Zimbra Collaboration Server. Some of the most significant statistics are in the LMTP Delivery Rate which was approximately, 5.22 messages per second injected, 13.38 messages per second received (multiple recipients), which implies heavy mySQL writes. The mySQL DB ( /opt/zimbra/db/data) after the last test was over 24GB on the larger of the two mailstores virtual machines. Despite what would be considered an extremely high level of activity, with numerous hard failures, none of the above data integrity checks suggested corruption or errors of any kind. Through all of the testing we were constantly attempting to induce failure of the Zimbra configuration. None of these attempts were successful at introducing any corruption into any of the Zimbra components. In all cases, the Zimbra SoapGen client performed consistently against the Zimbra servers without causing any corruption to any of the Zimbra components.

Zimbra Collaboration Server is an enterprise class messaging and collaboration solution designed entirely from open source. With its Web-based client, the Zimbra messaging and collaboration platform provides full functionality and access to all of its features regardless of the web browser used. In this study, we depicted a Zimbra 10,000 mailbox enterprise based on vSphere and NetApp NFS storage technologies. Using the 5,000-user virtual Zimbra building block this enterprise configuration is easily scaled to 100,000 mailboxes. Through our tests we have concluded that the Zimbra Collaboration Server when virtualized using vSphere and NFS datastores can provide a highly scalable and reliable messaging solution. Even under very heavy profile usage the solution performed as expected and we were not able to induce failure to either the storage or user mailbox data. The use of the VMware vSphere platform allows consolidation of physical machines by utilizing the existing virtual infrastructure to provide enterprise level email. By virtualizing the enterprise Zimbra deployments, the datacenter footprint is significantly reduced, thereby reducing power consumption and cooling costs, further reducing the operating expenditures (OPEX). The vSphere platform can provide the following benefits for virtualized enterprise Zimbra deployments: VMware vMotion Provides the capability of performing ESX host maintenance without interrupting for end users. Planned ESX host maintenance can be scheduled anytime of the day or night without email outage or interruption to end users. VMware DRS Can increase the performance of virtualized Zimbra deployments by intelligently placing Zimbra virtual machines on ESX hosts that can provide the necessary CPU and memory resources required by the Zimbra virtual machines. DRS can also increase virtual machine performance by balancing resources within the vSphere cluster. Significant gains of up to 35 percent have been realized through the use of DRS. See the VMware Web site at for more information. VMware HA Provides simple out-of-the-box high availability for the Zimbra virtual machines. In the event of ESX host failure, Zimbra virtual machines will be restarted on another ESX host within the vSphere cluster. The use of the NetApp NFS storage provides highly resilient storage to the vSphere and Zimbra virtual machine environment through the use of NetApp filer clustering. This environment provides fully redundant filer operations in any enterprise virtual environment. In the event of filer failure, the standby filer takes over functions of failed filer in seconds. NetApp storage can enhance the virtual Zimbra deployments through the use of the following features: Cloning Using the virtual Storage Console v2.0.1 (VSC) helps to automate virtual machine and datastore cloning. This can significantly reduce the time to provision virtual machines and datastores in the vSphere environment.


Virtualizing Zimbra with VMware vSphere and NetApp NFS

VSC also integrates datastore provisioning, resizing, deduplication and deletion. Using the rapid cloning utility (RCU) virtual machines can be provisioned significantly faster reducing virtual machine time to install. Thin provisioning Default provisioning option with NetApp NFS and VMware environments to reduce initial storage consumption. This feature allows oversubscription of a vSphere datastore for more efficient storage utilization and can actually reduce the need to provision additional storage. Deduplication NetApp block level deduplication allows VMware datastores to reduce the amount of storage utilization by the elimination of data duplication typically found on datastores due to many instantiations of guest operating systems. The elimination of this data duplication allows the VMware virtual environment to reduce the cost of storage as a result of better storage efficiency and utilization. Some of the following features of the Zimbra Collaboration Server allow for reduced TCO and a superior user experience compared to other messaging and collaboration applications: Folder sharing and document management Users can easily share files and folders throughout their organization. These features also provide the ability to manage the workflow in email. Archiving and discovery Simple to use email compliance that is built right into Zimbra Collaboration Server. No extra licenses are required to use this feature. Desktop synchronization Zimbra supports the use of other email clients such as Thunderbird, Microsoft Outlook to name a few, as well as the Zimbra Desktop client that also facilitates offline use. Contact management Personal and global address lists can be shared with ease across the entire messaging organization. Mobile access With the Zimbra Collaboration Server, users can accomplish over-the-air (OTA) synchronization to iPhones, Windows Mobile phones, BlackBerry phones, as well as access to the Zimbra mailbox using any mobile Web browser. These are just a few of the features that give the Zimbra Collaboration Server users an exceptional user experience compared to other messaging and collaboration applications. Zimbra Collaboration Server is the next-generation of email and collaboration. Zimbra can provide cloud and enterprise email and collaboration for any size organization. NetApp storage provides highly resilient NFS storage that can meets the demands of enterprise class virtual environments of any size. When virtualized with the VMware vSphere virtualization platform, you create a virtual application platform that is highly scalable, resilient and cloud ready.


VMware, Inc. 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001
Copyright 2011 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. Item No: VMW-TECH-WP-ZIMBRA-NETAPP-USLET-103