CommuniGate Pro Best Practices

Benchmarking

June 11, 2007

DNS environment: a good. it is recommended to setup an environment for testing. Following is a simplified overview diagram of a CommuniGate Pro Dynamic Cluster: Testing Environment Prior to installing and configuring CommuniGate Pro. fast DNS server (such as BIND 9) providing internal-only. “fake” domains for use in testing: Page 2 of 12 . This would include the following: 1.Best Practices: Benchmarking Introduction This document is provided by CommuniGate Systems Services as a helpful Best Practices guide to benchmarking a CommuniGate Pro Dynamic Cluster.

html Dynamic Cluster Overview The foundation building block of the CommuniGate Pro Dynamic Cluster is the “Shared File System” mounted on all CommuniGate Pro Backend Servers. Load generators: the test environment will include. including the load generators. the following systems for the benchmark load generators: a. It is critical to note that this file system must be shared (read and written) by multiple servers concurrently.communigate. 3.lan: a remote domain. which is called the “BASE” directory. SMTP sink – the postfix smtp-sink is a good. Please reference the following “Scalability” page of the CommuniGate Pro Guide for recommendations on tuning: http://www. at a minimum. some tuning may be required. Load generator – SPECmail or mstone. free tool The benchmark manager. Improper DNS setup will greatly decrease your test results.com/CommuniGatePro/Scalability. Generally speaking. Tuning: Depending on your test platform. A Cluster File System (CFS) on a Storage-Attached Network (SAN) System Layout Every CommuniGate Pro Frontend server performs all of its high-I/O operations in /var/CommuniGate. If located Page 3 of 12 . or others c.lan: your CGP “local domain” Remote. Benchmark manager – the main testing box. 2. could also function as other roles b. A Network File System (NFS) on a Network-Attached Storage (NAS) 2.Best Practices: Benchmarking • • Example. 2-4 is sufficient. It is critical that the DNS zones used in these tests include proper Arecords and PTR-records for all systems in the benchmark environment. load generators. This directory may be placed on a high-performance disk system for best performance. The Shared File System represents one or more file systems which are simultaneously accessible by all Backend Servers. The Shared File System can be of one of two types: 1. for relaying outbound messages to. and sinks can run on one box or many.

whether on NFS or when using a Cluster File System on a SAN. All Backend servers must have access to this shared filesystem.Best Practices: Benchmarking anywhere except /var/CommuniGate. can be placed on the NAS or SAN for some highest-level performance requirements). and is generally stored on the local disk (although. /var/CommuniGate CANNOT be shared across multiple systems – if locating /var/CommuniGate on the SAN. each system must have its own separate directory. Every CommuniGate Pro Backend server uses two main directory locations: /var/CommuniGate /var/CommuniGate/SharedDomains /var/CommuniGate provides the configuration functions for each system. then /var/CommuniGate should be a symbolic link to the real location. SharedDomains on NAS/SAN: System1: /var/CommuniGate System1: /data (Shared File System mount) System1: /var/CommuniGate/SharedDomains -> /data/SharedDomains (symbolic link) System2: /var/CommuniGate System2: /data (Shared File System mount) System2: /var/CommuniGate/SharedDomains -> /data/SharedDomains (symbolic link) Layout #2: Both the configuration directory and the SharedDomains on NAS/SAN: System1: /data (Shared File System mount) System1: /var/CommuniGate -> /data/CommuniGate-system1 (symbolic link) System1: /var/CommuniGate/SharedDomains -> /data/SharedDomains (symbolic link) System2: /data (Shared File System mount) System2: /var/CommuniGate -> /data/CommuniGate-system2 (symbo2ic link) System1: /var/CommuniGate/SharedDomains -> /data/SharedDomains (symbolic link) Page 4 of 12 . A symbolic link should be used to link this location to other real directory locations. /var/CommuniGate/SharedDomains must always be located on the shared filesystem. Here are two example layouts then: Layout #1: Configuration directory on local disk. However.

Depending on your systems’ true host and domain names. you want to configure your system something like the following steps.Best Practices: Benchmarking Installation Please reference these basic installation instructions here for specifics on installing on your specific platform: http://www. CommuniGate Pro Cluster Configuration Once you've gotten this far. this may be filled in correctly by default. such as “*. Under “Settings” section: 4. you will be prompted to enter a new “postmaster” password. 5.com” on each host (or similar. This password has complete administration rights to each system. and should be created securely. This is important.example. you log into the WebAdmin Interface (GUI) on each system at this location: http://<hostname>:8010 or https://<hostname>:9010 Upon initial startup of CommuniGate Pro. Cluster IP Addresses Settings->General->Cluster Add all Frontend and Backend IPs on all hosts: Example: Cluster Members Page 5 of 12 .communigate.example. Licenses Settings->General->Licenses Install all license keys on all CommuniGate Pro servers.html Once installed. if using a different domain). and once you’ve connected to the WebAdmin Interface. 6. This domain name entry MUST match the Dynamic Cluster license. for best performance. Domain Name Settings->General->Domain Name Set to “<hostname>. please remember to install all licenses on all servers.com”.com/CommuniGatePro/Install. All paths below are “clicks” within the CommuniGate Pro WebAdmin GUI.

1.sh For Backends: #### Startup.1.sh For Frontends: #### Startup.html For every server you are testing.168.3_____________________________ 192.168.communigate.2_____________________________ ________________________________________ ________________________________________ 7. Additional tuning recommendations for various operating systems for CommuniGate Pro are always available in the “Scalability” section of the CommuniGate Pro manual.Best Practices: Benchmarking This Server Cluster Address: On restart Active [first IP_____] [192. Add a “Startup” file on each server.sh SUPPLPARAMS=”—-ClusterFrontend --DefaultStackSize 131072 \ --useNonBlockingSockets --closeStuckSockets \ --CreateTempFilesDirectly 10” ulimit –n 16384 #### end Startup.1.1.com/CGatePro/Scalability.1_____________________________ 192. Stop the CommuniGate Pro servers on each host. available by default here: http://www.4_____________________________ ________________________________________ ________________________________________ Frontend Server Addresses: 192. you will need to add this file (default location): /var/CommuniGate/Startup.168.1.sh SUPPLPARAMS=”—ClusterBackend --DefaultStackSize 131072 \ --useNonBlockingSockets --closeStuckSockets \ --CreateTempFilesDirectly 10” ulimit –n 16384 #### end Startup.sh 9.168. 8.3] Backend Server Addresses: 192.168. Start all servers Page 6 of 12 .

d/CommuniGate start 10. A Cluster Controller is required before any other servers can join the cluster: # /etc/init. 1. Server-Wide and Cluster-Wide Client IPs Be sure to add your “local” IP ranges to the Cluster-Wide Client IPs Settings->Network->LANIPs->Cluster-Wide 2. If you open the WebAdmin Interface on your Cluster Controller. This page will display the entire cluster: MONITORS->Cluster (Note: The Cluster monitoring page is also a great place to manage all of your systems from. Queue Settings->Queue->Settings->Message Queue->Queue Foldering = 100 Settings->Queue->Settings->Message Enqueuer->Processors = 50 Settings->Queue->Settings->Message Dequeuer->Processors = 10 3. Tuning optimizations You must set these settings on all servers for most of these options. DNS Resolver Page 7 of 12 . then use the system links on the MONITORS->CLUSTER page. you can then easily login to the WebAdmin Interface for all your systems). and will take affect for all servers within the cluster.Best Practices: Benchmarking Be sure to start at least one Backend server first. Cluster settings (as part of any “Cluster-Wide” setting) only need to be set on one server. the Cluster Controller will show in the log the rest of the service: http://<controller hostname>:8010 Monitor -> Logs -> Display Confirm that all systems join the cluster. as the first Backend server automatically becomes the “Cluster Controller”. If the restart was successful.

TextMailbox or MaildirMailbox One can use with two options . rather than the default 8100/9100. it would probably be best to set the HTTPU->Listener ports to listen on 80 for HTTP and 443 for HTTPS.different filesystems perform different with each. SMTP Receiving Settings->SMTP->Receiving->Channels=1000 Settings->SMTP->Receiving->Disconnect after: [5] errors and Deny Access for: [15 minutes] 6. 8. LOCAL Settings->LOCAL->other=50 7. Cluster-Wide Domain Defaults Users->Domain Defaults->Cluster-Wide->”Account Storage”>Foldering Method = Hashed 2 Levels Users->Domain Defaults->Cluster-Wide->”Account Storage”>Rename in Place =Yes Users->Domain Defaults->Cluster-Wide->”Account Storage”>Generate Index =Yes 9. MaildirMailbox format uses more small files and many more filesystem metadata operations. SMTP Sending Settings->SMTP->Sending->Channels=1000 Settings->SMTP->Sending->Channels/Host->Other=50 5. Access Settings->Access->POP=1000 Settings->Access->IMAP=1000 Settings->Access->PWD clients=50 # (set PWD channels to at least 5 * <number of systems in cluster>) Also note here.Best Practices: Benchmarking Settings->Obscure->Domain Name Resolver->Concurrent Requests=25 4. and ASPs Page 8 of 12 . if testing the WebUser Interface. and additionally can take advantage of single-instance message store (which especially benefits Enterprises. Educational facilities.

here. As Page 9 of 12 .Best Practices: Benchmarking where one message may be sent to thousands of recipients). A combination using MaildirMailbox in the INBOX and TextMailbox is also possible – this method can be easily scripted in provisioning scripts.spec. and SPECsip committees. We provide some details of one of these. For most Cluster File Systems and NFS.org) and a voting member of the SPECmail. SPECmail2001 SPECmail2001 is a standardized benchmark developed by messaging vendors and research organizations that measures mail server performance using a realworld workload. Domain setup is managed in the WebAdmin Interface at the location: Users->Domains Be sure to create all your domains as “Clustered Domains”. TextMailbox format uses bigger mailbox files but these files are indexed for high performance. Benchmark Tools There are many tools available for benchmarking and testing e-mail. calendaring. and in most cases it is recommended. at http://www. and voice services. TextMailbox format may perform better due to less overall metadata file operations and good indexing. When using the “TextMailbox” (default) format. SPECimap. which are configured in the “Settings->General->Other” settings page: Settings->General->Other->”TextMailbox Manager”>Concurrently used large buffers=”50” Settings->General->Other->”TextMailbox Manager”->Index Mailboxes larger than=”1024K” (1 Mbyte) Optionally: Objects->Domains->Account Defaults->Server-Wide->New Mailboxes=TextMailbox (or MailDirMailbox) Objects->Domains->Account Defaults->Cluster-Wide->New Mailboxes=TextMailbox (or MailDirMailbox) Domain Setup Finally. SPECmail2001. The default is TextMailbox. add your clustered domain(s). it is recommended to reduce the default mailbox index size and increase the concurrent large buffers. and proceed with adding accounts. CommuniGate Systems is a member of the SPEC organization "The Standard Performance Evaluation Corporation.

beating all previous records by 67% or more at 2.html The CommuniGate Systems Services department has used the SPECmail2001 application to test CommuniGate Pro implementations prior to going live in production. Since 2004. The purpose of using this application in validation testing is that the initialization and benchmark test stages generate significant mail delivery and access load on the mail system.Best Practices: Benchmarking well.5-million users. CommuniGate Systems Professional Services has regularly tested most new implementations with SPECmail2001 to ensure that the system is Page 10 of 12 . CommuniGate Pro is the current world-record holder in the SPECmail2001 benchmark.org/mail2001/results/mail2001.spec. These results provide a good overview of what a large SPECmail test looks like: http://www.

The SPECmail2001 application should be capable of running on any Sun Javasupported platform. and the other acting as the SMTP sink for outbound delivery. and runs best on a relatively recent version of Sun Java (1. it is capable of demonstrating that the initial design of the system is fundamentally solid. anti-virus) and plugin performance SPECmail Preparation SPECmail2001 is a Java-based application. and client workloads. and that the hardware used is stable and capable of providing sufficient performance. one acting as the SMTP/POP load generator.000 users. or stability issues Shared storage performance External authentication or performance problems with integrated applications Content filtering (anti-spam. These (minimum) two systems cannot be part of the CommuniGate Pro Dynamic Cluster.sun. at least two "load-generator" systems are required for SPECmail testing. The benchmark focuses on the ISP as opposed to Enterprise class of mail servers. The goal is to enable objective comparisons of mail server products. The benchmark characterizes throughput and response time of a mailserver system under test with realistic network connections. Other java-like implements (such as GNU Java) are not sufficient . and firewalls in order to verify not just the CommuniGate Pro system(s). based on the Internet standard protocols SMTP and POP3. disk storage. Solaris.org SPECmail2001 is a standardized mail server benchmark designed to measure a system's ability to act as a mail server servicing email requests. and Mac OSX.4. The types of problems which SPECmail2001 can discover includes the following: • • • • • • • Faulty or insufficient hardware DNS server errors or configuration problems Load Balancer mis-configuration Network performance.com.we highly recommend you download the latest version of Java from http://java.2 or later). but also the entire production environment.Best Practices: Benchmarking prepared for deployment. Generally speaking. should be placed in a location to route connections through any relevant loadbalancers. Windows. with an overall user count in the range of approximately 10.000. routing.000 to 1. including but not limited to Linux. routers. SPECmail2001 Overview from SPEC. Page 11 of 12 . and for best results. While SPECmail2001 is not able to completely duplicate the 'real world' environment.

org> as of August 15. 2005. Page 12 of 12 .Best Practices: Benchmarking SPEC® and the benchmark name SPECmail® are registered trademarks of the Standard Performance Evaluation Corporation. visit http://www. The comparison presented above is based on Standard Performance Evaluation Corp.spec. based on the Internet standard protocols SMTP and POP3. For the latest SPECmail® benchmark results.org <http://www.spec.org/mail2001.spec.'s benchmark that measures a system's ability to act as a mail server servicing email requests. Competitive benchmark results stated above reflect results published on www.