Redbooks Wiki Optimizing Lotus Domino Administration

Redbooks Wiki: Optimizing Lotus
Domino Administration

1
Table of Contents

Part 1. Getting Started ............................................................................................................... 8
1.1. Documentation ............................................................................................................... 8
1.1.1. Content to document ............................................................................................. 8
1.1.2. Server Details ........................................................................................................ 9
1.1.3. Architecture Overview ........................................................................................... 9
1.1.4. Security Standards .............................................................................................. 14
1.1.5. Backup Standards ............................................................................................... 14
1.1.6. Naming Standards............................................................................................... 15
1.1.7. Instructions for End Users ................................................................................... 17
1.1.8. Conclusion........................................................................................................... 18
1.2. Daily, Weekly, and Monthly Checklist .......................................................................... 19
1.2.1. General recommendations .................................................................................. 19
1.2.2. Actions................................................................................................................. 20
1.2.3. Conclusion........................................................................................................... 23
1.3. Security Checklist......................................................................................................... 24
1.3.1. Additional References ......................................................................................... 25
1.4. Domino Authentication Options.................................................................................... 26
1.4.1. Choosing the Domino Authentication Options..................................................... 26
1.4.2. SmartCards (1) .................................................................................................... 27
1.4.3. Lotus Notes and HTTP Password Synchronization (3)....................................... 27
1.4.4. LDAP (4 and 5).................................................................................................... 28
1.4.5. SPNEGO (6)........................................................................................................ 28
1.4.6. Lotus Notes and Operating System Single-Sign On (7)...................................... 29
1.4.7. Tivoli Directory Integrator (9) ............................................................................... 30
1.4.8. Additional Resources........................................................................................... 30
1.5. Agents and the Domino Administrator ......................................................................... 31
1.5.1. Agent Triggers..................................................................................................... 31
1.5.2. Determining Which Agents Are Scheduled to Run on the Server....................... 32
1.5.3. Agent Manager Settings That Affect Agent Execution........................................ 32
1.5.4. Full Text Searches and Agents ........................................................................... 33
1.5.5. Performance Impact of New Agents and Applications........................................ 34
1.5.6. Troubleshooting Problems with the Agent Manager Task .................................. 34
1.5.7. Writing an Agent .................................................................................................. 34
1.6. Expanding the Domino Domain ................................................................................... 35
1.6.1. Registering and Securing a New Server ............................................................. 35
1.6.2. Replicating Critical Databases............................................................................. 36
1.6.3. Mail Routing......................................................................................................... 39
1.6.4. Monitoring and Managing Multiple Servers......................................................... 40
1.6.5. Clustering............................................................................................................. 40
1.7. Design, Replication, and Mixed Releases: Avoiding Design Ping-Pong ..................... 41
1.7.1. Description of the Problem.................................................................................. 41
1.7.2. Preventing the Problem....................................................................................... 41
1.7.3. Working with Mixed Clusters ............................................................................... 43
1.8. Routine Maintenance Best Practices ........................................................................... 44
1.8.1. Fixup and Transaction Logging........................................................................... 44
1.8.2. Regular Maintenance .......................................................................................... 44
1.8.3. Database Corruption ........................................................................................... 45
1.8.4. Hard Disk Fragmentation..................................................................................... 45
1.9. Mobile Access .............................................................................................................. 46
1.9.1. Traveler................................................................................................................ 46
1.9.2. BlackBerry ........................................................................................................... 48
1.9.3. Lotus iNotes Ultra-Lite......................................................................................... 50
1.9.4. IMAP/POP ........................................................................................................... 52
2
Part 2. Managing Users and Clients........................................................................................ 54
2.1. Optimizing Lotus Notes Client Administration Tips...................................................... 54
2.1.1. General Client/User Management Tips............................................................... 54
2.1.2. Recent Contacts.................................................................................................. 55
2.1.3. Local Replicas ..................................................................................................... 55
2.1.4. Smart Upgrade .................................................................................................... 56
2.2. Managing a User's Inbox.............................................................................................. 57
2.2.1. Using Inbox maintenance to manage mail file size............................................. 57
2.2.2. Quota Enforcement Options................................................................................ 58
2.2.3. Quotas with DAOS Enabled................................................................................ 58
2.2.4. Enforcing Quotas on Local Replicas ................................................................... 59
2.3. Policies......................................................................................................................... 59
2.3.1. Introduction to Policies ........................................................................................ 59
2.3.2. Using Policies to Standardize Secure and Simplify Your Environment .............. 60
Part 3. Effective Server Administration.................................................................................... 65
3.1. Monitoring..................................................................................................................... 65
3.1.1. Monitoring Options .............................................................................................. 65
3.1.2. What should be monitored? ................................................................................ 66
3.1.3. Monitoring Profiles for Domino............................................................................ 66
3.1.4. Domino Event Monitoring .................................................................................... 75
3.1.5. Further Reading................................................................................................... 79
3.2. Mail Routing ................................................................................................................. 80
3.2.1. Managing Spam .................................................................................................. 80
3.2.2. Mail routing and multiple directories.................................................................... 81
3.2.3. Journaling............................................................................................................ 82
3.2.4. Out of Office Notification...................................................................................... 85
3.2.5. Mail routing in a clustered environment............................................................... 85
3.3. Mass Mailings............................................................................................................... 87
3.3.1. The Mass Mailing Problem.................................................................................. 87
3.3.2. The Mass Mailing Solution .................................................................................. 88
3.3.3. Conclusion........................................................................................................... 94
3.4. Multiple Directories....................................................................................................... 94
3.4.1. Condensed Directory Catalog, Extended Directory Catalog or Directory
Assistance 94
3.4.2. Hints and Tips...................................................................................................... 95
3.5. Server Clustering Options............................................................................................ 96
3.5.1. Keep It Simple ..................................................................................................... 96
3.5.2. Redundant Domino Parts .................................................................................... 97
3.5.3. Domino Cluster.................................................................................................... 98
3.5.4. OS Cluster ......................................................................................................... 100
3.5.5. Internet Cluster Manager (ICM)......................................................................... 102
3.5.6. iNotes High Availability Configuration ............................................................... 103
3.5.7. IMAP failover (Domino 8.5 new feature) ........................................................... 103
3.5.8. Lotus Traveler Server High Availability ............................................................. 103
3.5.9. Load Balancer ................................................................................................... 104
3.5.10. Software proxy (IBM HTTP, nGinx, etc) ............................................................ 105
3.5.11. Sametime and QuickR High Availability............................................................ 105
3.5.12. Disaster Recovery Plan..................................................................................... 106
3.6. Transaction Logging................................................................................................... 106
3.6.1. General Transaction Logging Recommendations for 8.5.x Servers ................. 106
3.6.2. Transaction Logging Tips .................................................................................. 107
3.6.3. NOTES.INI Recommendations for Domino 8.5.x Servers ................................ 107
3.6.4. Domino 8.5.x and ODS 51 Updates.................................................................. 108
3.7. Domino Attachment Object Service (DAOS) ............................................................. 109
3.8. Managing Domino Indexing ....................................................................................... 111
3.8.1. View Indexes ..................................................................................................... 111
3
3.8.2. Full Text Indexes ............................................................................................... 112
3.8.3. Domain Indexes................................................................................................. 116
3.9. Backup a Domino Environment.................................................................................. 116
3.9.1. Backup Basics................................................................................................... 118
3.9.2. Backup Strategy ................................................................................................ 118
3.9.3. Backup Software ............................................................................................... 122
3.9.4. What to back up................................................................................................. 123
3.9.5. Backup procedures............................................................................................ 124
3.9.6. Backup Scripts................................................................................................... 126
3.9.7. Recommendations............................................................................................. 127
3.9.8. Summary ........................................................................................................... 128
3.9.9. Reference Reading............................................................................................ 128
3.10. Restore .................................................................................................................. 128
3.10.1. Disaster Recovery ............................................................................................. 129
3.10.2. Static File Recovery........................................................................................... 130
3.10.3. Domino Data File Recovery............................................................................... 130
3.11. Procedure to Restore Deleted Documents on IBM i.............................................. 136
3.11.1. Operating Type Save Restore Procedure ......................................................... 137
3.11.2. BRMS Full Save Restore Procedure................................................................. 138
3.11.3. BRMS Incremental Save Restore Procedure.................................................... 140
3.11.4. Additional Resources......................................................................................... 141
3.12. The Domino HTTP Server ..................................................................................... 141
3.12.1. General Server Configuration............................................................................ 141
3.12.2. iNotes................................................................................................................. 141
3.12.3. Troubleshooting and Tuning.............................................................................. 142
3.12.4. Additional References ....................................................................................... 142
3.12.5. Some Tips.......................................................................................................... 142
3.13. Domino HTTP Server Security .............................................................................. 143
3.13.1. Server Access ................................................................................................... 143
3.13.2. User Security and Authentication...................................................................... 145
3.13.3. Database Security ............................................................................................. 146
3.13.4. File System Security.......................................................................................... 150
3.14. Setting up a Redirection Application for Lotus iNotes users.................................. 151
3.14.1. Create the iNotes Redirect Application ............................................................. 151
3.14.2. Configure the iNotes Redirect Application......................................................... 152
3.14.3. Configuring the iNotes Redirect Application as the Default Home Page .......... 156
3.15. Securing Lotus iNotes............................................................................................ 158
3.15.1. iNotes and the Notes ID files............................................................................. 158
3.15.2. Active X Controls............................................................................................... 158
3.15.3. Browser Cache Management ............................................................................ 159
3.15.4. Encrypting Offline Databases............................................................................ 160
3.15.5. S/MIME.............................................................................................................. 161
Part 4. Tuning the Environment ............................................................................................. 162
4.1. Health Check.............................................................................................................. 162
4.1.1. High Level Checklist for Health Check.............................................................. 162
4.1.2. Performing the Health Check ............................................................................ 163
4.2. Document Configuration Tuner (DCT) ....................................................................... 172
4.3. Establishing a Performance Baseline ........................................................................ 173
4.3.1. Recommended Metrics...................................................................................... 174
4.3.2. Collecting Domino Statistics.............................................................................. 175
4.3.3. Reporting Database........................................................................................... 175
4.4. Domino Tuning Tips (all platforms) ............................................................................ 176
4.4.1. View Index Updates........................................................................................... 176
4.4.2. Disable Transaction Logging For Certain Databases ....................................... 177
4.4.3. Replication......................................................................................................... 177
4.4.4. Internal Caches ................................................................................................. 179
4
4.4.5. Multiple Mail Boxes............................................................................................ 181
4.4.6. Tips for Server Based Mail Rules...................................................................... 181
4.4.7. Tuning User Sessions ....................................................................................... 181
4.4.8. Domino Configuration Tuner (DCT) .................................................................. 182
4.5. Tuning for Virtualized Environments .......................................................................... 182
4.5.1. The Pros and Cons of Virtualization.................................................................. 182
4.5.2. Static Resources ............................................................................................... 183
4.5.3. Best Practices for Guest VM............................................................................. 183
4.6. Domino on Windows Tips .......................................................................................... 185
4.6.1. System Page Pool ............................................................................................. 185
4.6.2. Other Tuning Tips for Windows Servers ........................................................... 186
4.6.3. Additional references......................................................................................... 186
4.7. Domino on Linux Tips ................................................................................................ 186
4.7.1. Monitoring Server Resources............................................................................ 186
4.7.2. Operating System Limits ................................................................................... 186
4.7.3. Linux Services ................................................................................................... 187
4.7.4. TuneKrnl ............................................................................................................ 187
4.7.5. Troubleshooting and Debug Tips ...................................................................... 187
4.7.6. Disabling concurrent I/O and direct I/O on Domino servers on AIX.................. 188
4.7.7. Tuning Java for Domino on AIX ........................................................................ 188
4.7.8. Perfpmr for AIX.................................................................................................. 188
4.8. Domino on IBM i Tips................................................................................................. 188
4.8.1. Overview............................................................................................................ 188
4.8.2. Performance...................................................................................................... 189
4.9. Tuning Tips for the Domino HTTP Server.................................................................. 192
4.9.1. HTTP Server Threads ....................................................................................... 192
4.9.2. HTTP Requests................................................................................................. 193
4.9.3. JVM Heap.......................................................................................................... 193
4.9.4. Database Performance...................................................................................... 194

5
0.1 Preface

Note: This PDF document is the original text from the Optimizing Lotus Domino
Administration wiki found in the URL in which this document originated. Always
refer to the wiki version for the latest updates.

This IBM Redbooks wiki provides you with information on how to optimize Lotus Domino
administration. The focus is to provide Lotus Domino administrators with information on
how to get most of their valuable time.

Optimization of a Lotus Domino environment is not only a matter of how to set specific
configuration parameters on a server or on a client; it is more a conceptual approach on
how to address specific needs of the environment.

In this Redbooks wiki, we share our experiences and industry best practices about how
an optimized and smart Lotus Domino environment should look like and the checklists
and steps you should perform to ensure a smooth and optimized Domino environment.
Ideas and concepts presented here are meant to be an introduction, and are not meant to
be a complete list. If there are existing wiki articles, technotes, or whitepapers available
that have detailed discussion on the topics being presented we provide the reference
links.

Using this wiki

Each article in this Redbooks wiki is intended to be used stand-alone. However, you can
navigate back to the TOC and access all of the articles in the series by clicking the "Table
of Contents" link at the top of each article.
6
0.2 About the Authors

This book is produced by a team of specialists from around the world.

Paul Band is a certified IT specialist with IBM Software Services for Lotus in the
UK. He has been working with Lotus software within IBM for 10 years, having
started with Domino Development in R4.6. Paul has worked on countless customer
engagements ranging from troubleshooting customer issues to designing global
enterprise collaboration environments. In his spare time, Paul tries to improve his
golf swing.

Thomas Hampel is an IT Architect at IBM Germany. His key areas of focus are
migration projects, performance and delivery service architecture for customers
who outsourced to IBM. He is working with Lotus Domino since version 3 and is an
IBM Certified Advanced Lotus Developer as well as IBM Certified Advanced Lotus
Administrator in various versions up to 8.5. He is also an IBM Certified Advanced
Security Professional.

Amy Hoerle is an Advisory Software Engineer in the Lotus Support Center. She
has been focusing on Lotus products running on the IBM i Operating system for
over 10 years. She is also a frequent presenter at the COMMON, A Users Group,
annual meeting. When not working, Amy spends her time caring for her children,
volunteering at their school, reading or working in her garden.

Gladstone Lang is a IT Specialist at ITD/SSO, Hortolandia/SP/Brazil. He is one of
the account focals for the Email and Collaboration Service Line working with Lotus
products. Before joining IBM in 2004, Gladstone used to work at an IBM Client and
Training Center.

Vladislav Tatarincev is the Technical Director and co-owner of CYONE.
www.cyone.eu. He has a Master of Computer Science from Latvian University. He
has been working with Domino from release 4.5, for more than 10 years. He is also
an IBM Certified Security Professional. Vladislav is the author of many freeware
tools for Domino. His key areas of focus for Lotus Domino are: Performance,
Traveler, Security. His hobbies include: diving, shark diving, wreck diving,
underwater archeology, and motorbikes.
Wei-Dong Zhu (Jackie) is an Enterprise Content Management Project Leader
with International Technical Support Organization. She has more than 10 years of
software development experience in accounting, image workflow processing, and
digital media distribution using C, C++, Java, and Lotus Notes scripts. Jackie holds
a Master of Science degree in Computer Science from the University of the
Southern California. She is a Certified Solution Designer for IBM Content Manager
and has managed and lead the production of many ECM redbooks and Lotus
Domino redbooks wiki projects.

7
Part 1. Getting Started

1.1. Documentation
How many times have you come across a project or new client, where you have to
migrate hundreds of databases, application, and servers? You assume that everything is
documented and has always been well documented. As you begin to work on the project,
you realize that the existing documentation does not contain what you needed or
expected. This article describes how to properly document your Lotus Domino
environment and what kind of documentation you should have based on the size of your
corporation. The article serves as a valuable resource as you begin to document your
environment or assist you in evaluating your current documentation.
Before you start, here are some questions to consider regarding your existing
documentation:

Where is the documentation stored in? Is there a defined place for all
documentation?
Have you ever inherited an environment from someone else? If so, is the document
about this environment up to date as you were told?

Documentation helps the current and the next administrator or technical project manager
to quickly understand the environment. Properly and up-to-date documentation helps the
continuation and growth of your environment.

Lets look at what documents should be created and what information should be
documented.
1.1.1. Content to document
What makes a good documentation? Just listing the number of servers and host names
is certainly not enough in all cases.
A well documented environment helps to identify problems more efficiently, resolve
tickets faster, and helps to find bottlenecks.
The following table lists the kind of information that should be part of your documentation
and what implementation size benefits most from this type of documentation.

What Content to Document Small Medium Large
Server details + +++ +++
Architectural overview + ++ +++
Naming standards (+) ++ +++
Instructions for end users (+) ++ +++

8
1.1.2. Server Details
The very first and basic documentation is to have the server configuration described.
Often, the Domino Directory is misunderstood to be online documentation of the server
configuration. However, there is much more to be detailed for example, to allow someone
who is new to the environment to understand the building blocks.

The server configuration is essential for upgrade projects because it can uncover pitfalls
such as incompatibilities before they would cause a problem.

For each logical Lotus Domino server in your environment, the following (but not limited
to) configuration elements should be detailed:
Server names and IP addresses
Platform details such as hardware, operating system
Lotus Domino version with fix packs or hot fixes installed (if any)
Additional Lotus products with name, version and patch level
3
rd
party software installed with information on licensing details, install instructions
and describing how to access the installation package
For this part of the documentation, it is advisable to use the server name as the primary
key, e.g. have one document per server where you keep track of the current server
configuration details.

The resulting document must be able to be referenced in other parts of the
documentation. That's why a Notes application is a good choice for this kind of
information. Document links can be used to cross reference information.
1.1.3. Architecture Overview
An architectural overview diagram is a graphical overview of your servers, clusters and
their inter-connections. The main focus is set on Lotus Domino servers and how their
logical communication paths like mail and replication or SMTP connections.

The main purpose of the architecture overview diagram is to communicate a simple, brief,
clear and understandable overview of the environment that the administrators are
working with. Most likely, this is a drawing, much larger than one sheet of paper, which is
the building of your environment.

This diagram should include:
Lotus Domino Servers with their name and primary usage type and cluster
membership (if applicable)
Mail routing connections
Replication paths
For one way connections like "Pull only", use arrows to point out the direction of the
data flow
Inbound and outbound SMTP connections
Incoming and outgoing 3
rd
party connections like ODBC or other type of connections
Major non-Domino systems which Domino is communicating with
9

Some recommendations for creating an architectural overview diagram are:
Do no try to put too much information into the same drawing. It is important to get the
concept rather than all the tiny little details at this point in time. Focus on the Domino
level, ignore details such as hardware and operating system, patch level, etc.
Do not use abbreviations without explaining them in (e.g.) a legend. Make sure when
you build this overview, others can understand it easily. Especially for complex
environments, it is a key element to a successful documentation because even the
person who created this overview might not remember all the details later.
Work with your application development team to get an understanding of the 3
rd
party
connections. There is a high chance that administrators do not know what developers
have done in the past. It is essential to get a full understanding of these inter-
connections to avoid problems when applying changes to the infrastructure.
Make sure to describe the type of connection, e.g. by using different colors for each
connection type. Include a legend within the drawing for more details. An example for
this legend is shown below.
Keep this overview up to date by making sure changes in the infrastructure are reflected
in the drawing as soon as possible. In best case, the documentation is updated as part of
the change implementation.
Tools and Method to Create an Architectural Overview Diagram

To create a drawing, any vector oriented graphic software is acceptable. Even CAD
software or simple bitmap oriented graphic software will work. A broad number of
software products are available on the market, from commercial software products to
freeware applications. The product choice is yours. Just make sure to be familiar with its
usage and be comfortable with the licensing details.

Some examples for applications which are free of charge are listed below.
Dia
http://projects.gnome.org/dia/
yED
http://www.yworks.com/en/products_yed_about.html

After you have selected the software of your choice, create a drawing in the following
way:

1. Create an overview of the Lotus Domino Domains and outline how they are
connected. For this first step, the Domain can be represented by a cloud icon or
similar.

Review the replication topology within the Domain by looking into the Replication
Topology of each Domain. This information can be retrieved from the Lotus Domino
Administrator client within the Replication \ Replication Topology tab as shown on the
picture below.

10

Note: The Domino Administrator client will retrieve this information from the Domino
server which runs the maps task. This task is not automatically started on a Domino
server. For details on how to enable the maps task, refer to the Lotus Domino
Administrator Help http://www.ibm.com/developerworks/lotus/documentation/domino/

Add the administration server of each Domain and walk along the replication path.
Ensure that in the end of this process, all servers of a Domain were added.

Example

Here is an example of how a simple architectural overview diagram can look like:

11
Standards
Especially in large organizations, it is important to describe standards that apply to the
entire corporation. These standards can apply to every single detail of the environment,
sometimes predefined by other people in your company e.g. your operating system
standard or similar. Even if there are no regulations, it is advisable to define simple but
effective software standards, giving you and your peers the opportunity to work towards a
common target.
Hardware Standards
Start with documenting the hardware type and size, and used for what purpose.

From the Domino point of view you start by Defining server classes, this is where you
defer the server usability according to registered users or user access, location size,
server main task, etc.

For example, in Company A, the architect has defined that:
Small Servers should not exceed 150 users or host application for small locations (up
to 150 users);
Intermediate Servers can work as administration/application hub and should not
exceed 1000 registered users or host application where the concurrent users should
not exceed 1000 users;
Large Servers should not exceed 3000 registered users or host application where the
concurrent users should not exceed 3000 users. They may also be used for high
performance infrastructure servers (e.g. central SMTP gateways).

In a second step, for each of the server classes above, define:
Server Parameters: This is where you define based in the Server Class how your
servers configuration are going to be; type and amount of CPUs; how much memory
they should have, etc.
Hard Disk Layout: In here, you identify how your servers hard disk is or should be
configured, where would the operating system be installed; where the binaries are
going to be; and where is your data configured. Also in here, you should determine
which type of disk array you are going to be using or used according to each different
Server Class.
Be aware that vendors change their server hardware models quite often. New and more
powerful servers are being offered while the older ones might not be available anymore.
This is why you should consider the hardware standards as a rough guidance for new
Domino administrators or people who are not familiar with Domino itself. Keep them
updated on a yearly basis and do not hesitate to brainstorm smaller adjustments, e.g. to
use a more powerful CPU if it becomes available.
Note: Domino is a very I/O intensive application. When you choose a server model,
choose the system with best I/O thruput for best performance!

12
Software Standards
An important part of the environment documentation is the software standard. You
determine what will be the server operating system according to each Server Class.
Again, we can see that when you determine the Server Class correctly, it facilitates
everything that comes after.

Below is the suggested software standards that help you in your documentation:
Operating System (OS): You determine what should be the OS version and which
service pack needs to be installed according to the products requirements.
Depending in your Server Class, you can determine a specific OS.
Anti-virus Software: For anti-virus software, you have to differentiate between anti-
virus software for the OS and anti-virus software for Lotus Domino:
Anti-virus software at the OS level: You document the software version, patch level
applied, how the virus pattern files are being updated (how often they should be
updated), and also the files to be ignored by the OS anti-virus.
Anti-virus software at the Lotus Domino level: You document the software version,
patch level applied, and how the virus pattern files are being updated (how often they
should be updated).
Lotus Domino server: Document what is the Lotus Domino Server version and Fix
Packs applied according to each sub-software requirements. We all know that for
every product, there is a recommended Lotus Domino Server version with a specified
Fix Pack, which is very important to follow. Also, you document what should be the
folder naming convention for your Lotus Domino Server binary folder and data folder
Backup software: This is a tricky area. In many circumstances in a large company,
the Domino administrators do not work with the backup administrators. It is very
critical for every environment to have a good backup standard and policies very well
defined:
Domino server backup tool and policies: Document what are covered and how your
Domino server backups occur (daily, weekly, monthly) and what type (Incremental,
Full, the use or not of Transaction Logging Archiving).
OS backup tool and policies: Document what are covered and how your OS backup
occur (daily, weekly, monthly and what type for each Incremental and/or Full).
Special backups: If your company follows some sort of rule that request a longer
retention period or any special tasks that need to be done with the Lotus Domino
Server backup, this is the place where you should document it.
Environment monitoring: You document how your environment is being monitored
(which tool, and what if being monitored at the OS level and at the Lotus Domino
Server level).
Server reporting: You have to differentiate between reporting of operating system
specific data and reporting of Domino specific data. This document only covers the
Domino related reporting;
OS settings: Document the mandatory OS settings for all Domino servers. These
mandatory settings are necessary to support a stable and secure environment and to
minimize the support efforts.
Document what and how your OS should be set (e.g. drive letter, volume name, if
Windows update should be automatic or not, network card naming convention, registry
keys for OS tuning, etc):
Drive Indexing: If option is turned off as per Lotus recommendation (for each drive) or
if the service is turned off in the control panel (which covers all disk drive).
13
System Page file: Document how your systems page file is set.
Time settings: To assure a smooth mail routing and replication between all Domino
servers, it is important that the servers have been set to the correct time.
Time zone: The operating systems time zone has to be set to the correct value
according to the physical location of the server. In addition, the setting "Automatically
adjust clock to daylight saving changes" should be enabled and Domino servers
should be set to "use OS time zone settings".
Regional settings: Document how your servers Regional Settings are set, when
working on Windows OS environment;

Not all of the information must be described for a small environment. In large and growing
environments with multiple administrators and teams working in different time zones, it is
clearly a benefit to have common settings lay out.
1.1.4. Security Standards
One of the key components in a documentation is a description of the security standards
in your environment.
This information is not only essential in case of a disaster, its also supposed to clearly
describe whats allowed and whats not.
Describe the different roles and the limitations for each role.
E.g. User Administrator, Database Administratror, Application developer
Defined server access level, e.g. how the server security tab is supposed to be set
up.
Part of this is already described in the naming standards, so the focus should be on
the concept rather than the naming as such.
Limitations and permissions within the environment defined in form of a corporate
policy
Think about who is allowed to run agents, who is allowed to sign applications, script
libraries, etc. Have you got a specific role defined who will perform this work?
Handling of user id's and passwords, where are they stored, who (which role)
Handling of access deny groups
Where backup copies of key system (cert.id, server.id's, administrator id's.) files are
located and who got access to them in case of an emergency
Please note that the list above wont be a full list of items to be considered - additoinal
elements may apply to your environment, so please ensure to update this chapter of your
documentation on a regular basis.
1.1.5. Backup Standards
Documentation should also include a description of your environment is backed up and
what users can expect from a restore. This should include:
Backup policy, descrbing your corporate rules for how long a backup is retained
"What" explaining what kind of information is backed up how often. Here the focus
should be on Domino data and not so much on other elements such as the operating
system
"When", backup jobs are starting and when your backup window ends.
"Where", describing which servers do perform which role within your backup concept,
e.g. if one server is backing up data for all servers, etc.
14
"How", describing what backup software is used, where its installed
Details about the restore process and an estimate of how long it will take to have the
restore available
For more details about backup concepts for Domino please refer to the chapter
"Backup a Lotus Domino Environment".
1.1.6. Naming Standards
Naming standards allow people who are new to the environment to understand how to
maintain consistency while extending or changing elements within. The following list
represents configuration details of a Lotus Domino environment where naming standards
could be applied:
Server names
e.g. Define when to use what kind of server name.
As the previous example, Company B uses CCNNNNXXX where CC stands for ISO
country code, NNNN server Type (MAIL, HUB, APPL, etc), and XXX stands for the
server sequential number according to its Type.
Domain names.
Cluster names.
Domino named Networks.
Access control lists.
Directory names.
e.g. Describe the directory structure of a Domino server and where to put which type
of applications.
Note: You should document your company standards for servers directories (folders)
names for system databases, users Mail database, mail-in databases, and
application databases.
File names.
e.g. Define how to get a unique file naming in place.
Document the naming convention used by your company for file naming convention
at the Domino side. You can cover the naming convention for: users mail database,
mail-in database, common Domino application database, Domino application based
in the default templates; etc.
Group names.
Document the naming convention used by your company for groups used in the
Domino environment. Make sure to define groups following Domino standards: multi-
purpose, access control list only, mail only, server only, and deny list only.
Also, clearly define which characters are allowed and which are not.
Rooms and resources.
For information, see
http://www.ibm.com/support/docview.wss?rs=899&uid=swg21251010
User names.
Describe the naming convention used by your company for users names naming
convention. Cover all the details that are in use in your environment, such as: First
Name, Last Name, Full Name, Short name, Internet Address and any other important
field related to user that is in-place at your company.
e.g. Which characters are allowed or how and when to use middle initials.
Email addresses.
e.g. How an email address is computed if users can choose any address of their
choice or if there are restrictions within your organization.
15
Agent signer names
If you use specific ID files to sign or run your agents, then describe what syntax to be
used for them.
Note: There are limitations defined by IBM Lotus for each of the elements listed. For
more detail, refer to IBM Technote 1091216.
Access Control List

In this section of the documentation, include information on the database access control
lists (ACLs). They provide a flexible mechanism for the database owner to determine the
rights that individuals and groups have with respect to a Notes database. The ACLs for a
database are server specific, in that each server enforces the ACL as specified in its
version (replica) of the database.
For ACL documentation, include how your companys default template ACLs are set and
how the ACL of the system databases are set.
With this kind of information, you can improve your level of standardization and fine tune
your monitored environment.
To document ACL settings, we provide the following suggestions as how this can be
done:
Define basic ACL rules, e.g. Define ACL entries which must be present in all
applications or must be present in all mail files.
This can be all the groups and default IDs that should be in all databases ACL.
Normally, these are groups that are related to IT Administrator and Administrative
Server groups such as Domino Administration Process server, Server Backup, and
so on.
Databases that are under investigation. In most companies, the legal department can
request access to any database that may be part of some sort of litigation process.
Create and document the standards for this scenario.
Define how the ACL for Domino System databases should look like in your
environment.
Include every ACL entry with its access rights and roles from the following system
databases:
names.nsf, admin4.nsf, catalog.nsf, log.nsf, certlog.nsf, events4.nsf, ddm.nsf,
domlog.nsf, domcfg.nsf, busytime.nsf / clubusy.nsf, report.nsf, statrep.nsf, mail.box /
mailX.box (where X stands for the number from 1 to n depending on the amount of
mail.box set in the configuration document), statmail.nsf, mtstore.nsf, and da-CC.nsf
(where CC stands for the naming convention used for your companys Country code
or Site code).
Format to be used to document naming standards

It does not matter what format you use to document naming standards; however, it is
important that this part of the documentation is available to everyone. It must be simple
and easy to access and should be versioned.

There are multiple options to achieve this. Some examples are:
Stored in a Notes application, e.g. A discussion database where all users have
READER access.
Create a file based document (e.g. PDF) and store it in (e.g.) Lotus QuickR or any
other existing document management system.
16
An internal web page, where users can access the information by using their web
browser.
There are other methods. Use your favorite ones. Again, make sure the result is
accessible for all your users who need the documentation.

The following recommendations should be kept in mind when detailing naming standards:
In small implementations of Domino, naming standards most likely do not need to be
defined because the number of servers and domains are relatively consistent.
Nevertheless, it might be useful to define some naming standard to set the scene for
future growth.
Within larger Domino environments, corporate naming standards are defined or need
to be defined to define precise rules and limits for a team of administrators. This is
why it should include all the elements we mentioned in their naming standards.
Especially in large environments, allow users and IT people to improve these naming
standards by offering a discussion forum where people can ask why a certain
standard is defined in this way. Keep answers accessible for others and be open-
minded for changes suggested by your users.
1.1.7. Instructions for End Users

To reduce the number of help desk tickets and service requests, end users need to be
provided with a completely different kind of documentation. Most likely, end users do not
want to know about the detailed server configuration or their replication paths. Instead,
they look for guidance on how to manage standard requests in an efficient way.
Descriptions of common processes such as How to reset request new User ID or how
to reset a password is what end users are usually looking for.

The list of processes can be endless. Always aim to reduce the time support staff needed
to spend for answering the same question over and over again.

Consider this part of the documentation to be one of the first places for people who are
new to your organization. Its purpose is to be used for self assistance platform or a form
of guidance for your end users:
Process descriptions
What-if scenarios
Background information and cross reference (e.g. to naming standards)
Help desk phone number(s)
Additionally, keep the following recommendations in mind:
Include information for 1
st
level help desk, e.g. where to route tickets or how to reach
self service portals.
Do not mix end user instructions with training material. If people are new to Lotus
Notes, there are better existing resources to refer them to. You do not have to create
your own training material.
For more information please take a look into:
http://www-01.ibm.com/software/lotus/training/
http://www-01.ibm.com/software/lotus/training/multimedialibrary.html
17
Define the language which is to be used.
Depending on your corporation and regional distribution, not all users might
understand English.
The language is defined by what users understand best, not by what administrators
would like to use.
Extend this part of the documentation as needed.
Whenever you experience a growing amount of questions or requests in a certain
area, add one more instruction and cross reference them where needed.
Do not send mass mails to your users communicating how a new process looks like.
Instead, put the process description into this part of your documentation and refer
users to it by sending a link to the respective entry.
Format to be used for documenting end user instructions

Although there are many different ways to provide information, not all method provide the
required functionality:
Simple to access,
e.g. via web browser or Lotus Notes client
Distributable to a broader population
Accessible even when having a problem
e.g. Stored locally on the users desktop

One advisable method is to set up a new Lotus Domino application based on the Help
template. IBM Technote 1164526 describe how to do this.

Keep in mind that process descriptions are likely to change. Especially in larger
organizations, they are not easy to categorize because the processes differ e.g. between
countries, regions, business units or even between departments.
1.1.8. Conclusion
With a well documented environment, any infrastructure changes and any emergency
situation can be faced with more efficiency and more professionalism. It is important to
understand that documentation is not a static document. It is a living document with your
environment and needs to be updated on a regular basis to keep its value.

18
1.2. Daily, Weekly, and Monthly Checklist
Lotus products typically run smoothly. To ensure that they continue to run smoothly,
administrators should be equipped with the right set of tools for analyzing problems when
they occur. This article provides considerations and recommendations on what
administrators can do to improve efficiency or optimizing their system.
1.2.1. General recommendations
As a general recommendation, administrators should always be equipped with the latest
knowledge in the Domino area, understand how to analyze system performance, and be
able to troubleshoot server crashes.
Knowledge
One of the very basic elements is to know about capabilities of an environment, know
about bugs or problems, and practical methods to improve system functionality.
Optimizing an environment requires continuous improvement and fresh ideas. However,
fresh ideas are not always written down in a book yet. So you should always keep track
of the up-to-date information. We recommend few hints to help you or an administrator to
do this:
Sign up for My Notifications within the IBM Support portal.
With My Notifications you can receive daily or weekly announcements through e-mail,
custom Web pages and RSS feeds. These customizable communications can
contain important news, new or updated support content, such as publications, hints
and tips, technical notes, product flashes (alerts) and downloads and drivers. The
tool allows you to customize and categorize the products you want to monitor and
any of the available delivery methods to suit your support needs. http://www-
01.ibm.com/software/support/einfo.html

Since the early beginning Lotus Domino administrators have been collaborating with
each other to exchange information, best practices, hints, etc. One of the key elements is
to share knowledge with the community. This is why the biggest source of information is
the community itself. Make use of this valuable information by:
Signing up for the IBM Developerworks Lotus Community to read about Lotus
software products and strategy from those who develop it.
The Lotus Blogs are brought to you from IBMers who focus on Lotus software.
http://www.ibm.com/developerworks/lotus/community/
Actively participate in the product discussion forums.
Do not hesitate to ask your questions and feel free to answer questions from other
people.
ttp://www.ibm.com/developerworks/lotus/community/
Reading the blog posts of the Domino community.
This following site provides a good start because it consolidates Blog posts from
19
various Lotus community blogs. You may want to use the RSS Feed Reader
embedded in your Lotus Notes client in order to stay up to date.
www.planetlotus.org
Getting in touch with other Domino Administrators in your region by attending
community meetings or round tables.
Setting up your own Blog and post about your experience with Lotus
products.Encourage the community to provide feedback by registering your Blog at:
www.planetlotus.org
Performance
Troubleshooting performance related issues is not a simple task. Usually, these kind of
problems do not show up over night. Most likely, it is a phenomenon which develops over
time and this is also the reason why a performance analysis must take into account that
the root cause may be located outside of the Lotus Domino ecosystem.
Different tools and techniques apply to different platforms or areas that you want to focus.
IBM already provided a detailed article about performance best practices. To get an idea
of which tools you need for your platform, take a look at
IBM Technote #7008849 - Notes Domino Best Practice: Performance
http://www-01.ibm.com/support/docview.wss?uid=swg27008849
Server Crash Analysis
For troubleshooting server crashes, the following tools are valuable to have. Be familiar
with their capabilities, functionality and usage.
Analyzing server and client problems sometimes requires taking a closer look into
NSD files created by Lotus while a crash occurred. By default, these files are stored
within the Data directory of the client or server. Administrators can set up an
Automatic Diagnostics database to collect these NSD files in a central database
which will help to quickly get access to these files. For more information, see:
IBM Technote 1085850 describes how to set up this Automatic Diagnostic Database
IBM Technote 1272213 provides the latest version of the utility.
IBM Support Assistant is a desktop software which helps to find answers or solve
problems for various different IBM software products. It supports multiple IBM
products, not just Domino alone.
http://www-01.ibm.com/software/support/isa/
Lotus Notes Diagnostic is a Lotus Notes application which can analyze NSD files,
search for known problems and collect suggestions for a possible resolution of the
problem.
1.2.2. Actions
To help administrators in maintaining smooth Domino operations, optimizing server
stability, performance, and security, we provide recommendations for
daily/weekly/monthly tasks that are to be carried out.
Actions outlined in the sections below represent general best practice, but do not include
maintenance activities such as compact or fixup tasks which typically run scheduled.
Focus is set on what administrators are required to do to keep the Domino environment
healthy.
20
This is not an all-in-one perfect list as individual actions vary based on your environment.
If any of the described actions is performed, but does not result as expected,
administrators need to investigate further as there may be undiscovered problems. On
the other hand, not every action needs escalations or emergency actions. So even if you
are not investigating further, it is highly recommended to at least document an
abnormality with date and time of when the event occurred followed by comments or
clarifications.
Actions can be automated or may even be part of an existing monitoring solution. In this
case, the task shall be understood as task to verification of functionality.
To get an understanding of efficiency, where valuable time is spent, we recommend
documenting how much time administrators spend for which reoccurring action and
activity. After a certain period of time, the reported hours should be reviewed to identify
where (e.g.) small tasks take up a lot of the administrators time. In return, a conclusion
would be to evaluate if such a task or action can be automated in some way to optimize
your environment.
Daily Actions
The following list represents the daily actions an administrator should carry out for
optimizing server runtime, performance and security:
Check and resolve problems reported in Domino Domain Monitoring database
(DDM).
The type and number of issues shown are depending on your DDM configuration.
Check if there were any servers crashing.
If there is a problem, find the root cause by analyzing NSD files using tools
referenced in the general recommendations section earlier.
Check available free disk space.
This daily check can be automated by creating an event in a properly configured
Domino Domain Monitoring database (DDM).
Verify daily backup jobs ran successfully.
How to do this depends on the backup software used in the environment. All backup
software vendors provide log files which administrators can check. Some can report
success and failures through mail or other notification methods.
Check if anti-virus software is running properly and patterns are up to date.
Keep in mind to include the operating system anti-virus software in this check
because it is as important as the anti-virus software on Domino servers.
Check for replication problems by reviewing the replication log of your server(s).
The amount of time spent for this action can be minimized by setting up DDM
replication monitors which only reports failures to DDM.
Monitor mail routing by checking mail routing queues.
Check key statistic values on your servers and compare them with values from past
days.
For the number of peak transactions per minute, a fixed limit can not be provided
because the capabilities depend on the underlying hardware. Over time, you will get
an idea of when workloads become an issue and you will get to know how to balance
workload.

For the daily check, focus on these statistics:
Server.trans.PerMinute.peak
Replica.cluster.WorkQueueDepth.avg
Replica.cluster.WorkQueueDepth.Max
21

Note: This list does not claim to be complete. Depending on your environment, additional
or fewer actions may apply e.g. caused by third party tools which either requires
administrators to manage them separately or which help them to perform some of the
tasks mentioned above automatically.
Weekly Actions
In addition to the daily actions, administrators should perform the following actions on a
weekly basis, preferable in the beginning of a week to review the previous weekend:
Monitor Administration Requests database (Admin4.nsf),
Check the views Errors by Date and Individual approval required and take
appropriate action.
Review Domino server statistics and statistic trends. Especially search for workload
peaks and document them.
Take actions if you can see an unexplainable peak by reviewing log files further and
explore options to balance workload within your environment.
For more details, refer to Notes/Domino Best Practice Statistic and Events
Clean up your server and remove restored NSF files which are no longer required on
your server.
Typically, a restore is only required to be kept for a certain period of time, e.g. 7 days
after this time, the restored file can be removed from disk.

Once again, additional actions may apply to your environment.
Monthly Actions
In addition to the weekly actions, administrators should perform the following actions on a
monthly basis:
Monitor Domino server memory consumption and take actions to provide sufficient
memory for the Domino server to avoid performance problems. The total amount of
memory required depends on more than just the Domino server alone. Third party
tools on the operating system level that consume additional memory must also be
taken into account.
Check for new releases or patches affecting your environment.
Sign up to mailing lists as described in General Recommendations above.
If any changes are applied to the infrastructure, update the documentation to reflect
the current environment.
Run Lotus Domino Configuration tuner against all servers in the environment and
review recommendations made by DCT.
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/domino-configuration-tuner
Check Certlog.nsf for expiring certificates and ID files,
If required initiate the recertification process as described here http://www-
01.ibm.com/support/docview.wss?uid=swg21326765
Check server disk fragmentation, because fragmented disks may result in slow
server performance.
Note: You can run defragmentation tools on a server, but Lotus Domino must be
down to avoid damage to your data!
For more details, see IBM Technote #1229817
22
Review problems (if any) that have been reported in the previous month. Dig into
problems and try to find and fix the root cause.
For Domino Web servers, monitor web server requests.
The following IBM technotes can be helpful when working with Domino HTTP logs:
#1382231 - How can I gather and sort data from HTTP access?
1161104 How to reduce the size of Domino Web Server Log (domlog.nsf)

As mentioned previously, this list does not claim to be complete. Depending on your
infrastructure, the focus may vary and additional actions may apply.
Yearly Actions
In addition to the monthly actions, administrators should perform the following actions on
quarterly or yearly basis:
Perform tape backup restoration tests to ensure valid recovery data.
Just checking backup logs and reacting on errors alone is not everything. There is
nothing worse than a restore which cannot be done because (e.g.) backup media is
broken or it is empty for whatever reason. Once in a while, such as on a quarterly or
yearly basis, backups should be verified by restoring to a full server. The restore can
be done on an isolated server to avoid effect on your production environment.
Update your documentation.
We recommend updating documentation whenever a change is applied to the
infrastructure or at least with only a small delay. A quarterly or yearly update is
recommended to verify essential parts are correct. The larger an environment is, the
more frequent administrators should schedule this task.
Perform a server health check of your environment.
For more information on this topic, refer to the IBM OpenMic Call Health Check for
Lotus Domino servers in IBM Technote #1432995
1.2.3. Conclusion
In this article, we recommended a set of actions you should perform on daily, weekly,
monthly, quarterly, and yearly basis. These are not complete lists of actions. Depending
on your specific environment, come up with your own lists to perform on regularly basis,
to ensure smooth Domino operation and optimized system performance.

23
1.3. Security Checklist
According to some estimates, 80-90% of threats come from current or ex-employees.
This article describes activities which administrators should perform on a regular basis to
keep their environments secure. Server security consists of different areas, such as
server physical security (server room), operating system (OS) access by OS
administrators, and services that run on this machine. The security level of a server is
defined by the weakest node in the chain.
The following checklist guides you on what needs to be checked regularly to keep your
servers in good shape from a security perspective:
Check for unneeded account by enabling the License Tracking in Configuration
document. For small and medium environments, check every six months. For large
environments, check quarterly. According to statistics, about 10% of the accounts in
large environments are not used or not closed properly. For more information, see
License Tracking
Run ACL analysis on all databases. Check to make sure that there is no high Default
access, such as Default=Manager or Designer.
Also, if your server is accessible from the Internet make sure that all databases have
an Anonymous entry.
Note: Catalog.nsf may not have all databases. It lists existing entries. It does not
show that Anonymous entry is missing. You can write a simple LotusScript agent to
check the ACL or use the Domino Administrator client to perform a mass ACL
update.
Consider enabling LOG_USERSESSION=2 server notes.ini parameter as this will log
the IP address of the PC from which the user accesses the Domino server.
Ensure that the Domino web server is logging requests. You may filter some
resources from being logged. For more information, see License Tracking
Check that the Domino web server logs for attacks and scan your web site for brute
force password attempts.
If you have HTTP access to your server, consider deploying SSL for authorization so
that passwords are not transmitted in plain text over the network.
Check DDM database daily for new tickets. Consider enabling Security Probes in
Lotus Domain Monitoring. For more information, see Security Probe
Make sure that no person records have attached USER.ID files. You can run a
scheduled script to check this. In the case that the attached USER.ID is found, the
script should notify the administrator. Many organizations have default start
password. If password checking is not enabled, old copies of USER.ID may be used
for the wrong purposes.
Check if deny access group is updated and is populated in server documents.
Check for unneeded tasks running on server. For example LDAP running on public
server with Anonymous access allowed.
Check that only needed ports are open. Ask firewall administrators which ports are
allowed from outside (Internet). Make sure only needed ports are open. If a port is no
longer needed, close it on Domino and at the firewall level. Every opened port is a
potential way to get inside your system.
If you plan to do vulnerability scanning with third-party software, do this outside of
working hours. Notify administrators who are responsible for this system when you
plan to perform the test.
24
Check against information theft. There are third-party solutions that allow you to
check if anyone is accessing unauthorized data. There are Data Leakage Prevention
systems that can protect you against information theft.
Ensure that the Domino server has Internet password locking feature enabled. If
somebody does a brute force attack on a server, you can see this in the internet
lockout database. For more information, see Securing an IBM Lotus Domino Web server:
Using the new Internet lockout feature
Consider implementing stronger and more complex passwords. Do this step by step.
If a users password does not comply with policy, the user will be asked to change
the password. If the user cancels the password change procedure, Lotus Notes will
notify the user that the current password is not complying with policy and the client
will close.
Review the Security tab of your servers. Check who can enable Full Access
Administration mode, who can sign scripts that has server operating system access,
etc. Enable notification for enabling Full Access Administration to others, or a special
mailbox. For more information, see Technote # 1197579
Keep in mind, security can impact system performance and user experience. The more
secure the environment, the harder for users to access data to perform their work. You
should find a balance between the needed security level required by the business holders
and user comfort.
1.3.1. Additional References
For additional references and reading, see:
Lotus Security Handbook
http://www.redbooks.ibm.com/abstracts/sg247017.html
Security Considerations in Notes and Domino 7: Making Great Security Easier to
Implement

25

1.4. Domino Authentication Options
This article describes the Domino authentication options to help you determine which
option best suits your environment.
1.4.1. Choosing the Domino Authentication Options
Traditionally, Lotus Notes use USER.ID file for authentication. Release 8.5.x added some
new options for authenticating as well as a great feature that significantly reduces time for
ID management ID VAULT. Authentication using SmartCards is another option.

The following flowchart helps you understand which Domino authentication option is best
suited for your environment.

Each authentication option listed and numbered in the flowchart is described in the
following sections.
26
1.4.2. SmartCards (1)
SmartCards provide high security for Lotus Notes access. Regular Lotus Notes USER.ID
files can be stored on SmartCards and be protected with a PIN number. Every time you
log into Lotus Notes, you need to enter the PIN number, so the SmartCards can unlock
the protected USER.ID. You can also protect SERVER.ID files with SmartCards as well.
In that case, every time a server is started you need to enter the PIN number to unlock
the SERVER.ID.

If you decide to use SmartCards with Lotus Notes ID, you need to know how to configure
Lotus Notes/Domino for SmartCards usage. Keep in mind that SmartCards users may
require a separate policy. This is because periodic password changes and some other
options are not applicable for SmartCards protected USER.ID files.

For more information, see the following articles:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/enabling-smartcards-for-notes-login/
http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.notes85.help.
doc/sec_smartcard_t.html
==Single-Sign On (2)==
If you have many Domino Web servers, then Single-Sign-On (SSO) based on
LTPAToken can be used. When a user connects and authenticates to the first server, the
browser receives a secret ticket (token) that is stored in the browser. If you need to
authenticate to a new server, the browser will pass this ticket to the server (limited to
servers inside your Domain) and you will be authenticated without an additional password
prompt. For example, you have 5 servers (3 mail servers in a cluster, and two
applications servers, one of which is an internal application server and the other one is an
external application server). In this case, if you do not use (SSO), you have to enter your
password several times on each server. If SSO using a LTPAToken is used, you log in
only once.
Be cautious when you configure your server. If you use Internet Sites, then you use one
LTPAToken definition. If do not use Internet Sites, then you may use another LTPAToken
definition, which is stored in another document. It is recommended that you check the
($WebSSOConfigs) view for duplicated documents and that on all servers you use or not
use Internet Sites documents. This may save you time while deploying SSO.
You may also later use LTPA SSO during deployment of an Instant Messaging
(Sametime Entry) or Sametime Meeting Server. For example, you can configure the
Lotus Notes embedded Sametime client to log into Sametime using the SSO token.
Thus, you can eliminate the need for Lotus Notes users to enter an HTTP password to
log into Lotus Sametime.

For more information, see the following articles:
Can Sametime work with Internet Sites enabled?
Configuring Single Sign-On between Lotus Quickr and Lotus Sametime
http://www-10.lotus.com/ldd/lqwiki.nsf/dx/configuring-single-sign-on-between-lotus-
quickr-and-lotus-sametime
1.4.3. Lotus Notes and HTTP Password Synchronization (3)
27
Lotus Notes provides an option for users to set the HTTP password same as the Lotus
Notes password. The advantage of setting the same password is that the user has one
less password to remember. If the user uses the same password for both systems (Lotus
Notes and HTTP access), there is no need to spend time to set the HTTP password. It
happens automatically if it is set in Security Settings and added to Domino policy. When
needed, user can submit request to change the user's HTTP password.
Lotus Notes and HTTP password synchronization can be the first step to make
authentication easier for the users. This also helps reduce the number of help desk calls.
Lotus Notes and HTTP synchronization is available in Release 6.x, 6.5, 7.x, 8.x, 8.5.

For more information, refer to Security Setting help for enabling Lotus Notes and HTTP
password sync

Note, password synchronization does not work with Shared Login.
1.4.4. LDAP (4 and 5)
Lotus Domino users are registered in NAMES.NSF - the Domino directory. If you deploy
one more system, you may need to maintain multiple user accounts for single user in
your environment. User and password management can be a costly task in each system.
You can use Lotus Domino server for authenticating other users with the help of LDAP
protocol. Other systems may use Domino users for authentication with LDAP. The benefit
of this approach is if something is changed, e.g. password is changed or a new user is
added, all systems that use Domino for authentication automatically reflect the changes
and and see changes. You do not need to register or remove the user from different
systems.

For more information, see IBM Infocenter, Planning LDAP Features
1.4.5. SPNEGO (6)
Lotus Domino 8.5.1 introduced a new way for user authentication: Users can authenticate
in Domino web server with their Windows credentials. The benefit of the SPNEGO
authentication is that users are not asked to enter passwords. According to research,
password management in systems is rather expensive. Reducing the number of
passwords users need to have helps decrease the number of help desk calls.
If you currently use pre 8.5.x Release of Lotus Notes and plan to upgrade, consider
migrating from Single-Sign-On service to 8.5 so you can use Shared Login because this
solution does not synchronize password, thus it is easier to administer.

SPNEGO means Simple and Protected GSSAPI Negotiation Mechanism. The clients
browser and server can negotiate and the server can get information from Windows
Active Directory regarding which user is trying to access the system. In this way the
server will map the Windows user to the Domino users. If you consider adding SPNEGO
authentication to products such as Microsoft Quickr, Sametime, consult IBM Consultants.
Check requirements for Windows Active Directory and the Domino version before you
decide to deploy this.

28
For more information, see:
Deploying Windows single sign-on for Web clients (SPNEGO) in an existing Domino
environment
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Deploying_SPNEGO
Who supports SPNEGO authentication in a Lotus Quickr Domino 8.1 or 8.2
environment?
1.4.6. Lotus Notes and Operating System Single-Sign On (7)
Lotus Notes prior to 8.5.x offered an option that enabled users to not enter their Lotus
Notes password. Lotus Notes Single-Sign-On service was synchronizing the password
between Lotus Notes and the Windows operating system.
If you currently use pre 8.5.x Release of Lotus Notes and plan to upgrade, consider
migrating from Single-Sign-On service to Shared Login because this solution does not
synchronize password, thus it is easier to administer.

For more information, see
Does Notes support Single Logon with Novell?

==Shared Login (8)==
Notes Shared Login (NSL) is a feature introduced in Release 8.5. It allows you to unlock
your Lotus Notes ID with your Windows credentials. If the person is logged into the
Windows operating system, a special Windows service is responsible for unlocking Lotus
Notes USER.ID and the user can log into Lotus Notes without a password prompt. If the
user forgot his/her password, you need to reset his/her Windows Domain password.
Lotus Notes policy security settings in Release 8.5 has options on how to notify and
enable Shared Login for users. If you have enabled Lotus Notes and HTTP password
synchronization and you later enable Shared Login, users will now have to manage their
HTTP password separately. If needed, for some users, you can enable Shared Login in
the security preferences of the Lotus Notes client (grayed out by default).

29

Tips: Do not mix Shared Login with Single-Sign-On from Release 6.x-7.x. Single Sign-On
from old versions of Lotus Notes was synchronizing the Lotus Notes and Windows
passwords, Shared Login in 8.5 does not have to synchronize the passwords. A special
Windows Service UNLOCKs the USER.ID of the user. If you are upgrading from an
environment which used operating system Single-Sign-On, it is recommended to move to
the Shared Login feature. Operating system Single-Sign-On is maintained for backward
compatibility.

Using Notes shared log in to eliminate Notes password prompts
http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.notes85.help.doc/
sec_nsl_desc_t.html
1.4.7. Tivoli Directory Integrator (9)
Users who have purchased Lotus Domino 8.5.x are entitled to use Tivoli Directory
Integrator to synchronize data with Domino. For more information, see TDI Entitlement

If Tivoli Directory Integrator (TDI) is used to synchronize data with Domino, you do not
need to buy additional licenses. If TDI is used to synchronize non-Domino data, you need
to buy additional licenses.

TDI allows integration of different systems and synchronization of data. For example, it
enables you to integrate Lotus Domino directory and Windows Active directory as well as
export or import users to and from textfiles or CSV files. TDI is a very powerful tool. You
will need to take some time to learn its architecture before getting started with it.
1.4.8. Additional Resources
For more information, see:
Robust Data Synchronization with IBM Tivoli Directory Integrator
Home page of TDI Users community You can find here online education and tutorial
materials.
Lotus Domino integration page
Tivoli Directory Integrator - integration with Lotus Domino

30
1.5. Agents and the Domino Administrator
As a Domino Administrator, you may consider agents are for developers to write and
handle. However; there are a number of things every Domino Administrator should know
relating to agents. This article provides the list of these agent-related topics including how
can an agent trigger, how to determine which agnets are schedule to run on the server,
full text searches and agents, performance impact on agents, and how to write an agent.
1.5.1. Agent Triggers
Agents can be triggered in a number of ways and can also run under a number of server
tasks including agent manager, server and http. When an agent is created, the developer
can choose how that agent can run. An agent can be triggered on an event such as being
called from a menu, a mail message being delivered or a document being created. It is
important to understand which task an agent will be running under in order to be able to
recognize the impact of an agent. If you see a sudden increase in CPU usage on your
server, you may want to be able to easily identify or eliminate an agent as the potential
cause. The list of triggers is as follows:
On Schedule: This type of agent will run on the server and on schedule that was
defined by the developer. The agent may run multiple times per day, weekly or
monthly.
Action menu selection: For this type of agent to run, a user must selects the agent
from the Actions menu in the Lotus Notes client. When an agent runs in this manner,
the code will run either on the client or in the server task depending on how the agent
was coded. Domino administrator does not get a log or see a list of agents triggered
in this way.
Agent list selection: This type of agent trigger is reserved for agents that will be called
by another agent or only ran from the designer client. These agents will run in the
designer client or within the server task that called the agent.
Before new mail arrives: This type of agent is triggered by the router task immediately
prior to the message being written to the Inbox of a mail or application database. This
type of agent will run under the router task.
After new mail has arrived: This type of agent is triggered by a new mail message
being written to the Inbox of a mail or application database. Once the message is
delivered, the agent is scheduled to run in 2 minutes. This type of agent is controlled
by the agent manager task; this ensures that if multiple mail messages are written
into the database within a short period of time, the server will be able to process
those new mail messages during one execution rather than running multiple times
within seconds.
After documents are created or modified: When a new document is created or a
document is updated, this type of agent will run. The agent will be set to run as a
scheduled agent under the agent manager task once a document has been created
or modified, but not more than every 30 minutes.

31
1.5.2. Determining Which Agents Are Scheduled to Run on the
Server
The Domino Administrator client provides an easy way to determine which agents are
scheduled to run on the server. In figure 1 you can see that there are 5 agents scheduled
to run. For illustration purposes, the agents are named with their trigger type. As you can
see, only the agents that will be run by the agent manager task are listed. The next
scheduled run time can be easily seen under the Next column. If you place your cursor
over the schedule, the server will then display information on the schedule for that agent.
"A Scheduled Agent on a Server" is set to run every 60 minutes. If you were to hover
over "An After documents are created" or "An After New Mail Arrives" agent, you will see
a listing of "Agent will not run" to indicate to you that the agent will be scheduled based
on another trigger.

1.5.3. Agent Manager Settings That Affect Agent Execution
There are several settings that determine how many agents can run on your server at
any time and how long they may run. You find these settings in the server document,
Server Tasks... Agent Manager tab as shown below in figure 2. For example, by default,
only 1 agent may run at a time during the day and 2 agents may run concurrently during
the night. Depending on the number of agents that need to be processed in you
environment, you can raise this number.

Why would you need to increase the number of agents to run concurrently? In the section
above, you saw how to determine which agents are scheduled to run. If you saw many of
those agents scheduled to run in the past, and they are not agents triggered by a
document update or mail message, then the agent manager task is not keeping up and
you may want to consider increasing the Max concurrent agents parameter for daytime,
nighttime or both. Another parameter to consider is the Max LotusScript/Java execution
time. If you have agents that take over 10 minutes to process, you must change this
parameter or the agent will be stopped by the agent manager task before it is finished.
This setting is a safety precaution against infinite loops and to ensure one agent does not
prevent all other agents from running.
32

1.5.4. Full Text Searches and Agents
An agent may often times need to search for information and then take action. Full text
searches are resource-intensive operations when a full text index is not maintained. For
more information, refer to 3.8 Managing Domino Indexing. To determine if any agent is
performing full text searches on an application without a full text index, you can search
your server logs for the following message:
Warning: Agent is performing full text operations on database
'application/application1.nsf' which is not full text indexed. This is extremely inefficient.

You can easily do this using the log analysis tool. To access the log analysis tool, go to
the Domino Administration client, Server... Analysis tab. In the right panel select
Tools Analyze Log... as shown in figure 3.

At this point, you should be presented with a Log Analysis window. From here, you can
click on the Words tab to enter a string to search. In this case, the word inefficient is
enough as shown in figure 4. You can then click OK to begin the search.

33

Once the search as finished, you can review the Log Analysis Results. Below in figure
5 is the desired result of the search No matching entries found.

1.5.5. Performance Impact of New Agents and Applications
It is a good idea to have a performance baseline before adding a new application or
agent to your environment. This helps you to determine the impact of that application or
agent in your environment. For more information on this, refer to 4.3 Establishing a
Performance Baseline.
1.5.6. Troubleshooting Problems with the Agent Manager Task
For troubleshooting, see technote 7002851 - Troubleshooting Script for Notes
Scheduled Agents (and Agent Manager).
1.5.7. Writing an Agent
After you become more comfortable with Domino administration, you may want to move
into automating processes or writing an application. To get started with LotusScript, you
can refer to the Domino Designer help which includes a Lotus Script tutorial or
LotusScript Self-Help Training Modules. Additional resources can be found in the Lotus
Notes and Domino Application development wiki.
34
1.6. Expanding the Domino Domain
At some point, you will most likely be in the position to expand your Domino Domain.
Whether this is to add a Sametime or Traveler server into your domain or add an
additional mail server; when you make your first transition from a single server to a
multiple server environment, there are some things you need to keep your environment
running smoothly. This article walks you through the process of expanding your Domain.
1.6.1. Registering and Securing a New Server
The first step when adding a new server to your Domino domain is to register the new
server. Similar to registering a user, the registration process will create a server
document and server ID file for the new server. The registration process will also
automatically add the server name to the LocalDomainServers group if it exists. At this
point, you can modify the server document to set the proper authorities for the new
server. In the example shown in figure 1, you can see that the server administrators have
been defined, the administrators have also been allowed the programmability restriction
of Sign or run unrestricted methods and operations. In the server access section,
database and replica creation has been restricted to administrators only and the
Terminations group has been denied access. You are now ready to configure the new
server using the configuration wizard appropriate for the platform you are using for this
new server.
35

1.6.2. Replicating Critical Databases
In a multiple server environment, there are 2 databases that must be replicated among
your servers names.nsf and admin4.nsf. This is critical in order for the administration
process to function correctly. In small companies with 10 Domino servers or less,
replication is typically a push/pull replication from the administration server as shown in
figure 2. Replication is scheduled using a connection document. In the example you can
see that connection is from Server1/Company A to Server2/Company A. On the
Replication/Routing tab, the replication type is Pull Push which means changes made in
those databases will flow to and from each server. The Files/Directory paths to replicate
has been defined as names.nsf and admin4.nsf so only those databases will replicate.
Finally, on the Schedule tab, the replication is configured to run every 20 minutes every
day.

36

For more information the administration process including example replication topologies
for larger Domino deployments and adminp basics, refer to the Notes/Domino Best
Practices: Administration Process article. Be aware that to prevent replication conflicts
and to ensure proper operation of the adminp task, the domino directory (names.nsf)
must have the same server specified as the administration server for ALL domino servers
in the domain. This can actually be expanded to any application. You will not have one
server in your domain that is the administration for every database in your domain;
however, each application replica should use the same administration server.

While only names.nsf and admin4.nsf must be replicated, you can choose to replicate all
databases in common between the servers. In this case, be aware that nearly all
templates will replicate between the servers. For more information, refer to the Domino
Template Replication and Design article on the Domino wiki. In most cases, this is
desired; however if you need to co-exist in a multiple release environment and do not
want the design to be updated to the latest release, see
7.5_Design_replication_and_mixed_releases__avoiding_design_ping-pong for more
information.

In figure 2 above, you saw an example of probably the most common and simple
replication configuration. There are a number ways you can improve on that schedule
when working in enterprise size deployments of Domino:
Set specific times in the schedule rather than an interval.
Set a replication time limit.
Avoid replicating databases during scheduled maintenance times.
Consider time zones.
Avoid pushing massive updates to the Domino Directory during prime business
hours, especially important for large enterprises or large directories containing a
hundred thousand users or more.
37

What is the difference between using the times and setting a time range and repeat
interval? The difference is that with the repeat interval the replication will not always
occur at the same time. That is because the repeat interval starts when the replication
completes. So if the first replication starts 12:00 and takes 15 minutes, the next
replication will occur at 12:35, not 12:20. When setting a specific start time, it can be
easier to troubleshoot replication if you know exactly what time it will occur. Set a
replication time limit to avoid one replication event from starting before the previous
replication is completed. Keep in mind that if you want very frequent replication on some
databases, it will be much simpler to use a repeat interval.

To consider these keys points, imagine that Company A has recently expanded their
enterprise globally and added a group of servers in Rio de Janeiro. The company now
has two groups of servers Server Group US and Server Group Brazil. They have decided
they want to replicate names.nsf, admin4.nsf and their mail files every 60 minutes. They
run their backups at 12:00 a.m., the design task at 1 a.m. and updall task at 2 a.m. They
run weekly compacts on all databases on Saturday from 4 a.m to 7 a.m. How can you
create a replication topology that satisfies the business requirements and avoids the
maintenance windows assuming these servers are located in the EST and BRST time
zones?
EST Time BRST Time Comments
12:00 a.m.
3:00 a.m. Daily
3:00 a.m. 6:00 a.m. Daily Nightly maintenance window
for US servers
4:00 a.m. 7:00
a.m. Saturday
7:00 a.m. 10:00 a.m.
Saturday
Compact for US servers
9:00 p.m.
12:00 a.m. Daily
12:00 a.m. 3:00 a.m. Daily Nightly maintenance window
for Brazil servers
1:00 a.m. 4:00
a.m. Saturday
4:00 a.m. 7:00 am. Saturday Compact for Brazil servers

You can now easily see what times you should avoid scheduling large replication events
depending on which server you initiate the replication. Assuming Server 1 is located in
the EST time zone, you can create a connection document as shown in the following
figure.

38
By creating the connection document in this way, you can accommodate for the
maintenance windows and time zones for both servers for every day but Saturday. You
can then create another document for Saturday. The following figure shows the example
for Saturday from the server 2 located in Brazil to Server Group US.

1.6.3. Mail Routing
Typically you want all of the servers in your Domino domain to be able to exchange mail
as needed. If all of your Domino servers are in the same Notes Named Network (NNN),
then Domino automatically know how to route mail between your servers. If they are not
in the same NNN, then you need to have two connection documents: server1 to server2
and server2 to server1.

What is the Notes Named Network? The NNN is a way to logically group your servers
based on your physical network. It is defined in the server document in the Notes
Network parameter which is found on the Ports Notes Network Ports tab in the server
document as shown in figure 6. By default, the first server in a Domino domain is created
into a network called Network1 and additional servers default to TCPIP Network. Thus
you should choose a value for your servers or set the network name to match if
appropriate based on your network topology.

39
In order to send mail between two Domino servers, the server must know where to send
the mail and must be able to make a connection to the remote location. If you have
properly configured your NNN or connection documents. you have satisfied the first
requirement. The second requirement requires a close working relationship with your
network administrator. The Domino servers uses the Notes remote procedure call
(NRPC) protocol over the port 1352 when sending mail or replicating. If you have a
firewall between your Domino servers, you must ensure that port 1352 is open in both
directions in order for mail to be successfully exchanged.
1.6.4. Monitoring and Managing Multiple Servers
Monitoring the log or Domino console manually is a rather straight forward task in a
single server environment. As you add more servers to your environment, this task
becomes increasingly more difficult. Domino provides multiple tools to simply this task so
it is important to remember to use these tools and add your new server to this
environment. Information on configuring monitoring and alerts can be found in 3.1
Monitoring.

Here is a simple checklist for you:
Add the new server to DDM Hierarchy.
Configure Diagnostic Collection for the new server if you do not use a global
configuration document for this.
Configure statistics collection document for new the server.
1.6.5. Clustering
Another common reason for expanding the Domino domain is to be able to provide high
availability to your users. To date, there is no hardware or software that will or can
guarantee the system will be up from now until the end of time. Therefore your Domino
server will be down at some point. Whether the server is down for routine maintenance,
upgrades or a natural disaster; Domino application clustering is simple to configure and
use to provide high availability for your users.

Key clustering concepts for the new Domino Administrators are:
Domino clustering is application level clustering.
Failover is limited to the Lotus Notes client. A browser accessing iNotes will not
automatically failover to a cluster server without additional infrastructure.
Mail can be routed to another server using Domino clustering. For more information
refer to 3.2 Mail Routing.
Another applications such as the resource reservation database can be clustered
using the natively available Domino clustering.
A Traveler server cannot be clustered.
For detailed information on clustering refer to the article Understanding IBM Lotus
Domino server clustering and 3.5 Server Clustering Options.

40

1.7. Design, Replication, and Mixed Releases: Avoiding
Design Ping-Pong
Do you have multiple Domino servers in your environment? Do you upgrade your Domino
servers over multiple days, weeks or months? If so, you should be aware of the options
available to prevent a problem where the design task and replication are changing the
design of your databases daily. This article explains to you how to avoid design ping-
pong scenario.
1.7.1. Description of the Problem
There are several ways you may notice that your servers are doing "design ping-pong".
You notice in your log that each night when the design task runs, all of your databases
are updated. Your users noticed that one day, their mail database was upgraded and the
following day their mail file is not upgraded any more. You noticed the replica task is
replicating more data than normal.

What can cause this behavior? It is the combination of replication and the design task. By
default, the design task runs at 1 a.m. on your server. By default, the line
ServerTasksAt1=catalog, design is in the NOTES.INI of every server. When the design
task runs, it compares the date and time of the design elements in the template with the
design elements in the database. If there is a mismatch, then the design task replaces
the existing design element with the one from the template. This is important so that
when you upgrade your Domino server, your databases get the latest updates and fixes.
Now imagine you have a Domino server you just upgraded to 8.5.2 that replicates with an
8.5.1 server. The following sequence of events may occur:

Day 1: Domino 8.5.2 server upgrades the design of all mail files and system application
databases with the latest 8.5.2 updates. During the next replication event between the
servers, the 8.5.2 design replicates to any replica databases present on the 8.5.1 server.

Day 2: The Domino 8.5.1 server updates the design of all databases with the copy of
design elements it has stored in the copy of its templates. During the next replication
event between the servers, the 8.5.1 replicates design to any replica databases present
on the 8.5.2 server.

Day 3: The cycle begins again; the Domino 8.5.2 server upgrades the design...
1.7.2. Preventing the Problem
There are a number of ways you can prevent design ping-pong from occurring:
Replicating templates between servers
Limiting the design task
Removing templates from some servers
Selective replication

41
Replicating Templates Between Servers
Most templates have the same replica ID between releases. If you allow templates to
replicate in your environment, then the latest design will get replicated to any templates
that are in common between the servers. For more information, refer to the Domino
Template Replication and Design Domino wiki article. In this scenario, since all servers
have the same design elements when the design task runs, it will recognize that the
database has already been updated and thus not update the database again.
Limiting the Design Task
By default, all Domino servers run the design task at 1 a.m. If you need to run in a mixed
release environment for a period of time and do not replicate templates, you can choose
to modify your server configuration and not run the design task on one or more servers in
your domain. The design task is started at 1 a.m. due to the ServerTasksAt1=catalog,
design entry in the server NOTES.INI file. You can remove the design task by entering
the following command at the Domino console: set config ServerTasksAt1=catalog.
Once you have upgraded all of your servers you can easily re-enable the design task
using the set config command again: set config ServerTasksAt1=catalog,design.
Removing Templates from Some Servers
The design task cannot update the design of databases if the template does not exist on the server.
In the scenario described above, the administrator can remove the templates from the
8.5.1 server. Later, as part of the server upgrade, the templates are copied back to the
Domino server data directory. Be aware that removing the templates can cause a
performance problem on the server if you have clients or servers configured to replicate
these files. For details on the potential performance impact of removing the templates,
review technotes 1299812 and 1426125.
42
Selective Replication
When replicating Domino databases and applications, you control which data can
replicate. For example, if you only replicate a small subset of databases, you may want to
temporarily disable replication of design elements. You can do this in the replication
settings of a database as shown in figure 1.

1.7.3. Working with Mixed Clusters
The approaches discussed in this article work great for replica servers, but they may not
work for clustered servers. That is because the cluster replicator always replicates design
elements. The easiest way to prevent the design from going to the old server is to not run
the design task on either server in the cluster. However; without the new design, you
cannot use the new features. Also system databases such as names.nsf and admin4.nsf
are backwards compatible so the new design will work fine on your older servers.
Because of the complexities here, it is best to upgrade your clustered servers together or
within a short period of time.

43

1.8. Routine Maintenance Best Practices
Having a weekly maintenance schedule is a crucial part of maintaining good health of
your Domino server. Domino provides a simple way to schedule maintenance tasks using
Program Documents in the Domino Directory. This article provides best practices for
routine maintenance.
1.8.1. Fixup and Transaction Logging
One of the benefits of transaction logging is that if a server is restarted following a fault
(server failure, power failure, hardware failure etc.), logged databases do not require a
consistency check before users can access it. After you set up transaction logging, Fixup
is not needed or used to bring databases back to a consistent state.

Therefore for transaction logged servers, the Fixup task should not be scheduled to run
regularly.
1.8.2. Regular Maintenance

Regular Maintenance of the Domino Directory is essential to maintain the health of a
server and optimize performance. Administrators can use the following tips as a starting
point to create a maintenance schedule tailored to their environment.

IBM technote 1248830 provides a short summary.
Weekly Reboot of Windows Servers
Many customers reboot their Windows servers on a monthly schedule to prevent memory
leaks.
Recover Database White space
When documents and attachments are deleted from a database, Domino tries to reuse
the unused space, rather than immediately reduce the file size. Sometimes Domino
cannot reuse the space or, because of fragmentation, cannot reuse the space effectively
until you compact the database.

The following command can be scheduled to run at the weekend using a Program
Document:
compact -b -s 10 (Archive Transaction Logging enabled)
compact -B -s 10 (Circular or ArchivedTransaction Logging disabled)

With the above command, any database that has more than 10% white space will be
compacted. The -b, or -B means that the server will perform an in-place compaction. The
b (lowercase) switch should be used with transaction logs so that it does not assign new
DBIIDs to the databases.

44
Note: New with 8.5.2, the compact -ODS switch performs a copy-style compact only if the
current ODS is less than desired default ODS. This is useful if you need to perform a
compact to implement new features (such as document data compression).
Compact Administration Requests database
The following command can be scheduled to run at the weekend using a Program
Document:
compact admin4.nsf -c -t
With the above command, a copy style compact is performed. This is useful because it
can solve database corruption (a new DBIID will be assigned). The -t switch will disable
transaction logging for that system database.

Note: If a new DBIID is assigned to a database, a full backup of the database should be
performed as soon as possible. This is because old transaction logs can no longer be
restored to that database. For more information about the database DBIID, read IBM
technote 7003909.
1.8.3. Database Corruption
There is no need to run fixup or updall as part of the weekly maintenance schedule. Fixup
should be run only if you suspect corruption in a database and updall is run every night
by default. You only need to run updall with switches if you suspect view corruption or
other such problems within the views.

If a database corruption is detected, run compact -c first. If the a view index has become
corrupted, run updall -r -t . The database corruption troubleshooting guide has additional
suggestions to minimize corruption within a database.
1.8.4. Hard Disk Fragmentation
SAN or RAID-5 based Domino data volumes can be affected by fragmentation. The
mechanical hard-disk is the slowest component of a server. Maintaining file fragmentation
can help maintain performance. The Hard Disk fragmentation Notes/Domino wiki article
explains in-depth hard-disk fragmentation and its effect on Domino servers.

Note: File fragmentation also affects workstations. Regular o/s de-fragmentation helps
maintain disk performance on users' client machines too.

45
1.9. Mobile Access
For mobile access, there are different options available for a variety of device types. This
article discusses various options and solutions available for mobile access.

The following table summarizes the supported device types and the available features for
each.
Device Type Traveler BES Server POP/IMAP iNotes Ultra Lite
Blackberry X
iPhone/iPad/iPod X X X
Android X X
Windows Mobile X
1.9.1. Traveler
Lotus Traveler is a Push-mail solution included in Lotus Notes licence. It is available at no
additional charge if you have purchased Lotus Notes 8.x or Lotus Notes.8.5.x.

It is recommended to run Traveler on a separate server. Traveler must installed on top of
the Domino server and it is called an Advanced Product (a product/component that is
installed on top of Domino). The version of Lotus Traveler you may install depends on
Lotus Domino version you server is currently running. Review the software requirements
to check which version is supported for this Traveler version you desire (for example,
Lotus Traveler 8.5.2 can be installed only on Lotus Domino 8.5.2). Additional information
on Traveler can be found in the following link:
http://www-01.ibm.com/software/lotus/products/notes/traveler.html

Lotus Traveler supports Windows Mobile, Nokia Symbian, Apple iPod/iPhone/iPad.
Android phone support is added in fixpack 1 of 8.5.2.

To deploy Lotus Traveler, you need:
1. Install Lotus Traveler on top of Lotus Domino.
2. Distribute Traveler clients on mobile phone (they can be downloaded from Traveler
homepage, or distributed remotely).
Installing Traveler Server

Lotus Traveler server is supported on Windows 32, Windows 64 and Linux OS. Linux OS
support was added in release 8.5.2.

Lotus Notes Traveler system requirements can be found in the following link:
http://www-01.ibm.com/support/docview.wss?rs=475&uid=swg27007909

Things to check before you start installation:
The Domino Servlet Manager is enabled. To do this, open the Server Document of
the Traveler server, Internet Protocols -> Domino WEB Engine tab. Set the Java
Servlets to Domino servlet manager.
46
Verify that the Lotus Traveler server is included in LocalDomainServers group so it
has Manager access to all mail files.
If your Traveler server is located in a DMZ, enable Configuration Directory. When
using a configuration only directory the Domino Directory will contain only
configuration files as all other information will be requested from other servers that
has a full copy of names.nsf.
It is recommended to have a SSL certificate on a server. This improves the security
of your server. Using a HTTPS connection ensures that no one can intercept (listen
to the network traffic) when a password is sent from the phone to the server. With a
standard HTTP connection the password is transmitted in plain text.
Traveler versions prior to 8.5.2 use two ports, one for AutoSync and the second for
data transmission. In 8.5.2, there is only communications on HTTP or HTTPS port.
For complete and updated information on things to check prior to installing Lotus Notes
Traveler, review the information from the following link:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Before_you_install_LNT8521

Installing a Traveler server on both Windows and Linux is quite easy. On Windows, the
LotusTraveler setup file is launched, and you choose what you need to install, see
options below. On Linux, the Traveler installation can be performed in two ways, via
graphical mode or silent install. The Silent install is preferred method since it does not
require graphical mode which maybe missing on some systems. Just configure a silent
install answer file and run installation with silent install switch:
Clients (installs only images for Nokia/Windows mobile/Apple)
Server only (install only server part without clients
Both (recommended option) - this will install both option, server and client

After you complete the Traveler installation wizard, pay attention to last screen of
installation wizard. Check to make sure the installation is completed successfully.

Lotus Traveler stores information about registered devices in lotustraveler.nsf database.
Do not delete device documents from this application, Lotus Traveler can be managed
only with console commands.
Troubleshooting
If you have issue with phone, try to understand if this is the problem for one user or for all
users. This helps you understand, where to look for solution on device or on a server.

The most popular problem is autosync is not working. In most cases restarting the phone
solves this problem.

If only one user has problems, try to login to Traveler server using a web browser with the
user's credentials. You may see additional valuable information that may help you as you
investigate the problem.

You may add additional logging for this device/user with Lotus Traveler console
commands. When the problem is solved, do not forget to disable logging, as this
generates debug XML files on server, and they fill disk space.

47
In case the entire server is not functioning, search the console for JAVA errors and
search for a solution from IBM support. Reinstalling Traveler is also an option.
Reinstalling does not require much time and it will fix problems with missing components
of Traveler.

For more troubleshooting information, refer to the following references:
Traveler 8.5.2.1 Server Troubleshooting:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Server_troubleshooting_LNT8521
Traveler 8.5.2.1 Client/Device Troubleshooting
http://www-
10.lotus.com/ldd/dominowiki.nsf/dx/Troubleshooting_known_limitations_and_restricti
ons_LNT8521
Traveler Frequently Asked Questions (FAQ) and Common Issues

For Lotus Traveler console commands information, refer to the following link:
http://www.lotus.com/ldd/dominowiki.nsf/dx/Console_commands_LNT8521
1.9.2. BlackBerry
If you are looking for information about Domino and you BES, consult this table as it
shows the Domino Release/Server OS and BES:
http://na.blackberry.com/eng/support/software/server_domino_ver_march_05_10.pdf
Domino Server Installation on a Windows Server Before Blackberry
Enterprise Server Install
A Domino Server installation is recommended on the server where the Blackberry
Enterprise Server will be installed. The installation makes the administration easier.

The installation should follow a usual Domino installation as shown below:
1. Check if the server document for the new server has been created and exists in the
Domino Directory.
2. Prepare for the Domino server build, placing the following files on the server in a
temporary location: Server ID file , Domino server executable file, and Domino system
files.
3. Install the Domino server code by running the self-extracting installation executable,
answering the prompt questions.
4. Copy or move the files from the temporary location to the permanent place.
5. Execute a compact in the system files by opening a command prompt. Use ncompact -
D filename.nsf (for more info about compact).
6. Configure the Domino server by double clicking on the server program icon, answering
the prompt questions.
7. Click on Domino server program icon to start the server for the first time. Server is
ready for installation of BES.

48
Preparing the Environment

To have the environment prepared, perform these steps (which help to prevent problems
in the future):
Check if the Domino Server is a member of the LocalDomainServer group in the
Domino Directory and if Adminp is running. Try opening admin4.nsf,
Check if the server can access the other servers in the environment.
Check if the setup application for the BlackBerry Enterprise Server can access the
Domino environment variables in the NOTES.INI file.
Verify that the remote DB2 or SQL Server is accessible. This step must be performed
if you choose to have Blackberry Manager to use authentication in a SQL or DB2
Environment.
To verify, type CMD and then type telnet 1433. If you have connection, the cursor will be
blinking. If that happens, close the Command Prompt because you can connect to server.

Post Blackberry Enterprise Server Installation
After you complete the BES installation and you want to be sure Domino is properly
running, check the console for messages such as Message sent to handheld, OTAFM
or OTAC. These messages indicate that messages are flowing from the server to the
environment.

Start the Blackberry Controller after the messages are being delivered that will prevent a
silent crash causing nbes and domino to restart on the server again.

If dispatcher is not starting up, it is necessary to check SQL or DB2 environment. In most
of the cases, the connection between the BES and the server is causing the problem.

There are some cases where a recycle of the SQL machine that holds the SQL/DB2
needs to be performed. NBES is related to the Domino task that starts the BES for
Domino. To start it using load nbes command at Domino Console and to finish it use
tell nbes quit command at Domino Console.

Blackberry Controller can be stopped while server is initializing users. It must be started
once all the users were initialized and mails are flowing properly. We can Stop/Start
anytime it to prevent a silent crash (unexpected reboot).

Locating a Problematic Blackberry State Database
Many times if a Blackberry server crashes it is due to a problematic end user Blackberry
state database. To find out where the problem is, you can do the following:
From the console log file, locate problem thread ID
[1114:000D-1384] Thread=[1114:000D-1384]
Stack base=0x07110090, Stack size = 12068 bytes
PANIC: LookupHandle: null handle

49

Do a search in NSD on "FATAL" to find the problem thread id to confirm.
If there are multiple crashes, several NSDs can also be checked to ensure it is the
same file causing the problem.
FATAL THREAD with PARAMETER DATA 12/143 [ nserver: 1114: 1384]
Search NSD on "Open Databases" and look for file with problem thread ID.
[Domino Install directory]\data\BES\state\123456789.nsf
Version = 43.0
SizeLimit = 0, WarningThreshold = 0
ReplicaID = 0x87256f00:0x004b5bba
bContQueue = NSFPool [ 000f6545]
Offline = No
DeleteInProgress = No
FDGHandle = 0xf0240409, RefCnt = 1, Dirty = N
DB Sem = (FRWSEM:0x0244) state=-1, waiters=0, refcnt=1, nlrdrs=0
Writer=[ nserver: 1114: 1384]
SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1, refcnt=1 Writer=[ nserver:
1114: 1384], n=0, wcnt=0, Users=-1, Owner=[ nserver: 1114: 1384]
By: [ nserver: 1114: 000d] DBH= 154, User=CN=COMPANY/NEWORG
From the information we get from the NSD, we see that the 123456789.nsf file
caused the server to crash.

Fixing a problematic Blackberry state database

You can fix a problematic database by doing one of the following options (you must
decide which one can be applied to your environment):
Deleting or renaming the .NSF file with the problem and recreating it.
Executing maintenance (fixup, compact or an updall) on the DB using the Domino
console.
For more information, tips and tricks from RIM Website:
http://na.blackberry.com/eng/support/blackberry101/tips/
1.9.3. Lotus iNotes Ultra-Lite
Lotus iNotes Ulta-Lite is for mobile devices and can also be with a traditional PC browser.
For reference, go to http://www-
01.ibm.com/support/docview.wss?rs=899&uid=swg21315871

Lotus iNotes runs on the following client operating systems:
Microsoft Windows XP Professional and Microsoft Windows Vista Business and
Enterprise Editions using the following browsers:
Internet Explorer or Mozilla Firefox
50
Novell SUSE Linux Enterprise Desktop 10 using the following browsers:
Mozilla Firefox
RedHat Enterprise Linux Desktop 5.2 using the following browsers:
Mozilla Firefox
Macintosh OS X 10.5 using the following browsers:
Mozilla Firefox
Safari 3.1.x
Apple iPhone and iPod Touch firmware version 1.1.4 or later (for the ultra-light mode)
There is increase in mobility request to have mail configured or available on devices
using a web browser. When enabling iNotes, confirm the following settings are
enabled/configured:
End user mail file:
ACL to Anonymous=No Access
ACL Advanced tab = Maximum Internet access to Editor.
Server Address Book:
Must be Domino 851 or greater
Forms85.nsf File
Must be the only Formsxx.nsf located on server iNotes directory. If not, it can cause
an end user issue when opening in the browser. Be sure to remove any old
Formsxx.nsf listed from old installations.

Remember that any change that you do with FormsXX (like removing old files)
requires you to reload HTTP on server (restart task http). If you do not execute this
command, it will give you a Read Only error when trying to open a mail file via
Internet Browser.
Directory Assistance File
This should be on the lastest template available.
SSL Configuration
Install the SSL certificate by copying the necessary files to Domino Data directory
(.kyr & .sth files).
Open your the server document for your server and go to the Configuration ->
Servers -> All Servers -> your server -> Ports -> Internet Ports tab and enter the
SSL Key file name. You should also verify the SSL port is enabled.
HTTP Config
Define the hostname and/or IP addresses to be used by this server. To do
this, open your Address book and go to Configuration -> Servers -> All
Servers -> your server -> Internet Protocols -> HTTP. Set the Bind to host
name field to enabled and set the Hostname(s) field to the proper host
names and/or IP addresses for your server.
51
Set a Home URL for the server, for more information see 3.14 Setting Up a
Redirection Application for Lotus iNotes Users. For example:
https://yourservername/homepage.nsf?Open

Ensure the operating system is not running any other http task that could
interfere with the Domino server. If it is, disable it before enabling Domino's
http server. If you do not do that, Domino will not be able to use port 80/443.

Verify HTTP and SSL is working properly:
Via the Domino Administrator client review the active tasks. Verify that the
HTTP task is running.
To verify a SSL connection is in use, access the server using this syntax ->
http://yourserveraddress and confirm that you see the little lock on the bottom
of your browser.

For more information, go to
http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.help.do
mino.admin85.doc/H_LOTUS_INOTES_ULTRALITE_MODE_STEPS.html

1.9.4. IMAP/POP

What is IMAP?

You can view just the heading and the sender of the message and then decide whether
to download the mail. You can also create and manipulate folders or mailboxes on the
server, delete messages, or search for certain portions or an entire message.

Domino IMAP users can:
Replicate messages from the server that runs the IMAP service and store them on
end user local replica.
Access messages directly from the server (different from POP3 users who download
the messages first).
What is POP3?

Post Office Protocol 3 (POP3) is old and less sophisticated e-mail protocol. When you
read your mail, all of it is immediately downloaded to your computer and it is no longer
kept on the server. This can be a problem if you want to access your mailbox on the
server or different computers due to possible hardware problems, virus, or preference .

52
The IMAP protocol is preferred over the POP3 protocol because if we enable both, we
can downgrade the performance of the server running Domino. Other considerations:
When using POP3, we will force the end user computer to connect to server, start a
push/pull to bring messages to the end user computer.
When using IMAP, the end user will synchronize the local replica with the server only
for new messages to avoid server to be in a sync mode for a long time which
consumes CPU of the server.
Other item that must be considered is that the messages should be stored in MIME in
order to prevent CPU issues.
For more information about performance, go to:
http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.help.domino.a
dmin85.doc/H_SETTING_UP_POP3_USERS_STEPS.html
http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.help.domino.a
dmin85.doc/H_ENABLING_A_MAIL_FILE_FOR_IMAP_ACCESS_STEPS.html

53

Part 2. Managing Users and Clients

2.1. Optimizing Lotus Notes Client Administration Tips
As a Domino Administrator, you know that in order for your users to be productive and
happy, the Lotus Notes client must be working properly. Keeping the client configured for
optimal performance and up to date is important. There are a number of tools available to
help you accomplish those goals. This article introduces you to these features and
provides you the access to the best documentation on these topics.
2.1.1. General Client/User Management Tips
Understand the installation options available to you. Clients may be installed
manually or deployed automatically. For more information refer to the Domino wiki
article Notes Client Deployment or the IBM Redpaper Distributing Notes Clients
Automatically. You can also customize the Lotus Notes installation, refer to
Roadmap: Customizing your Notes client installation.
Lotus Notes is extremely powerful software; however, it seems that users are often
not aware of how to use many of the features or realize they exist. End-user training
can resolve this issue. There are many available options using third-party vendors,
free videos from sources such as YouTube or the IBM Multimedia Library. There are
also a number of videos related to learning Lotus Notes available in the Domino wiki.
Starting at version 8.0, the concept of a "basic" client was introduced. The Standard
client is Eclipse based and has a heavier footprint from the previous Lotus Notes
client version. For more information, refer to technote 1264877.
Trying to understand what client type to use and which functions are available can be
confusing. The Domino wiki provides a feature comparison for the Lotus Notes
standard, Lotus Notes Basic, Lotus iNotes and Traveler clients to simplify this task for
you.
Use the Domino Smart Upgrade feature to simplify client updates. You can use smart
upgrade to install maintenance releases, fix packs and/or other patches. For more
information refer to the Smart Upgrade topic below.
Know your clients. Do you know what client versions are still being used at your
company? Not sure? You can use license tracking to help you answer this question.
Ensure users only have editor access to their mail files. This is important to prevent
users from changing their ACL permissions or granting other access as well as
forbidding them to accidentally delete their own mail file. Refer to the InfoCenter for
information regarding the access levels in the ACL. Note that users may still use the
delegation functionality within the mail and calendaring features with editor access.
Also, giving users editor access to the mail file on the server prevents the user from
creating a full text index on their mail file. This is desired to ensure all full text indexes
54
in your environment are created in a way to have the least amount of impact on your
server.
Did you know that you can automate application roll outs? If you are still sending
application links to users or manually creating local replicas for your remote users,
there is an easier way! Using policies you can add bookmarks or have databases
automatically replicated to clients. For more information, please refer to the 2.3
Policies article.
Managing unread marks can be confusing. For example, do you know unread marks
are stored in a table? Do you know unread marks are unique to a user and may not
replicate? To understand how unread marks work and options available, refer to
technote# 1140018.
Use a local replica and a mobile directory catalog to allow your users to send mail
and perform directory look-ups off-line. For information, refer to the information below
on local replicas and the wiki article 3.4 Multiple Directories.
Be aware of client issues affecting your users. Configure automatic diagnostic
collection (ADC) to work for you. With ADC, if the Lotus Notes client crashes, the nsd
and other relevant diagnostic data will be sent to the fault recovery database on your
Domino server for your review. Crashes can be easily monitored and categorized.
For more information refer to What is the Automatic Diagnostic Data Collection tool.
The maintenance tasks (fixup, updall, and compact) can be run in the Lotus Notes
client. The tasks are located in the client executable directory and the name is
preceded with an "n", for example nfixup.exe, ncompact.exe and nupdall.exe. The
switches are the same as on the server.
2.1.2. Recent Contacts
Starting with the Lotus Notes client version 8, the concept of Recent Contacts was
introduced. With Recent Contacts, anyone who you send/exchange mail with is
automatically added to your "recent contacts" list. In addition to Recent Contacts, the
type-ahead feature was enhanced. When addressing a message, type-ahead provides
results in order where the contact you interact more often with is at the top of the list. This
can be a good and be a very time-saving feature, it can also have some negatives. For
example, if someone you exchange mail with gets a new e-mail address, a new entry will
be added to your recent contacts list and the old address will remain. This can be
frustrating with the type ahead feature. You should delete the old address to avoid
sending emails to the old address. For more information on the recent contacts feature
including information on disabling this feature, refer to recent contacts FAQs.
2.1.3. Local Replicas
Using a local replica to access mail has become a more popular option over the years. It
is easy to understand why when you consider a properly configured client with a local
replica and directory catalog will provide users access to their mail from anywhere
connected or disconnected. Also, users working from a local replica require less server
memory and CPU to service compared with a user directly accessing their mail on the
server. This means you may be able to add more users to a server or postpone an
upgrade when moving from server based mail files to local mail files. When you hear this
55
statement do you immediately think "I don't want to go to each client?" or "My users are
going to complain because they won't see new mail immediately" or "If the mail is
replicated locally it is a security risk as anyone could read that data if the PC or laptop is
stolen". Fear not, Domino 8.5 takes away these concerns. Using desktop policies, you
can automatically deploy local replicas, configure a default replication schedule and
ensure the data is encrypted on the client. Even better, Domino 8.5.2 implements an
enhancement to local replicas with a new feature called managed mail replicas. With
managed mail replicas:
Users are automatically redirected to the server if the replica is unavailable for any
reason.
Not all mail has to be replicated to the client or mail can be retrieved from the server
"on demand."
Replication may be triggered immediately upon receipt of a message.

Simply put a managed mail replica eliminates the negatives of using a local replica. For
more information regarding using a server based mail file versus a local replica or
managed replica refer to the following article:
http://www-
10.lotus.com/ldd/dominowiki.nsf/dx/IBM_Lotus_Notes_and_Domino_8.x_local_mail_repli
cas_Advantages_considerations_and_best_practices

For more information regarding Managed mail replicas refer to:
Configuring managed replicas using the Desktop Settings document
Managed Mail Replica: Use Mail free of network delays and server outages
2.1.4. Smart Upgrade
Smart Upgrade is the process to notify the user to update his/her client to a later release
and perform the installation. As you learn about using Smart Upgrade, there is one key
point that is often times overlooked. Policies, while recommended, are not required to use
smart upgrade. Keep this in mind as you learn about and implement Smart Upgrade to
prevent issues with users being upgraded before you are ready.
The prerequisites for using Smart upgrade are:
A Lotus Notes Client (version 6 or higher) already installed
Connectivity to a Domino server
Smart Upgrade database
Smart Upgrade database defined in a Server configuration document.
For more information refer to:
Understanding Lotus Notes Smart Upgrade
Notes Client Deployment
Steps to create a Smart Update Database
How to deploy non-versioned patches via SmartUpgrade

56

2.2. Managing a User's Inbox
A higher number of documents contained within the mail file Inbox creates extra work for
Domino mail servers. The Update task must perform more processing to maintain the
sort order in the View Index when new mail is added, or old ones deleted. When one
considers the cumulative effect of many hundreds or thousands of mail files in large
organizations, Inbox management is a key factor in maintaining performance and stability
of the environment.

This article provides Administrators with some tips on how to keep control of an ever
growing estate of mail files.
2.2.1. Using Inbox maintenance to manage mail file size

The Inbox maintenance feature is designed to improve server performance by reducing
the size of users' Inboxes in mail files. This has the effect of reducing the View Index size
for $Inbox and also reduces the amount of work the Update task must perform.

When you enable the Inbox maintenance feature, the Administration Process periodically
runs the Inbox Maintenance agent on each users' home mail server. The Inbox
Maintenance agent resides in the mail template, MAIL8x.NTF. The agent removes
documents from the Inbox based on settings you define in the Server document or the
mail policy settings document. These documents are still stored in the mail file and can
be accessed via the All Document view.
There are two methods to enable the Inbox maintenance feature; both in the Domino
Directory (names.nsf). The first, is in the Server Document; Server Tasks ->
Administration Process tab. The second is in the Mail Policy document; Mail -> Basics
tab. The settings that are specified in the Server document override the Inbox
Maintenance settings in the mail policy settings document. Administrators should
therefore choose one method only, based on the following strategy:
Goal: Implement Inbox maintenance globally on the mail server (Use Server
document).
Goal: Implement Inbox maintenance selectively to a subset of users or groups (Use
Policy Document).

For further information, read Using Inbox Maintenance to manage mail file size, in the
IBM Lotus Domino Administrator Information Center.
Frequently Asked Questions

57
2.2.2. Quota Enforcement Options
For a beginner's guide, read Understanding Quotas for IBM Lotus Domino Mail
Databases.

Administrators have a number of options available to enforce (or not) mail file quotas.
The options available differ in strength of enforcement and help Administrators to
implement organization policy. Administrators can set two thresholds; warning level and
maximum level, and whether the maximum level is enforced. Notifications can be sent to
user's Inbox to give early warning that they are close to exceeding their mail file size
quota.

Administrators have three options of quota enforcement. These options are outlined in
figure 1 and instruct the mail router how to treat mail addressed to an over-quota user.
Over Quota
Enforcement
Description
Delivery Anyway Use this if quotas will not be enforced by the mail router. This is the
default setting.
Non Deliver to
Originator
New messages are not delivered and returned to the sender. They
are informed that the message could not be delivered because the
recipient's mail file was full.
Hold mail and retry New messages are held in the MAIL.BOX. Further options must be
set to refine behavior. Administrators need to be aware that using
this setting can lead to very large MAIL.BOX sizes if large messages
are held for long periods.

For further information, read Setting Quota controls for the router, in the IBM Lotus
Domino Administrator Information Center.

2.2.3. Quotas with DAOS Enabled
When calculating the size of a mail file to determine whether it conforms to configured
mail quota or warning threshold limits, the server treats attachments stored using the
DAOS as though each user owned the entirety of the attachment file. Therefore the full
size of the attachment in every message delivered to a mail file counts towards the mail
file quota. Likewise, when a user deletes a message, the full size of the message is
removed from the mail file quota.

The actual file size of the mail database that uses attachment consolidation therefore
does not necessarily reflect its logical size. For example, a user's mail file might exceed
its quota limit of 100MB even though the physical size of the file is only 65MB. IBM
Technote #1405456 discusses how DAOS functions with Mail Quotas.

Administrators familiar with earlier versions of Domino may have viewed the Space Used
column in the Files tab of the Domino Administrator. The Space Used column gave an
indication of the actual amount of data stored within the NSF file, as opposed to blank
space. Note that Domino 8.5 saw the last of the Space Used Column. For further
information, read the DAOS and The Space Used Column wiki article.

58
2.2.4. Enforcing Quotas on Local Replicas
If a quota is set on a server-based replica, by default the quota is not enforced in a local
replica and neither are quota warning notifications displayed. Administrators can
configure their environments so a quota warning is displayed when users attempt to
create a new mail message or calendar invitation on a local replica.

IBM Technote #1247798 provides a comprehensive guide to the procedure for
implementing this feature.

2.3. Policies
Have you looked into Domino policies yet? If not, then you have not yet seen how
powerful policies can be. Since there are many reasons to use a policy and a number of
policy types, you may wonder how you should get started with policies.
2.3.1. Introduction to Policies
The purpose of a policy document is to assign settings to users. There are a number of
available setting types in Domino 8.5.x. including Registration, Setup, Desktop, Security
and more. For a list and definition of each setting type refer to the Domino Infocenter
Policies topic.

There are two types of policies:
Explicit
Organizational
An organizational policy is a policy that will be applied to the entire organization or
organizational unit. For example, if you work for Fictional Software Company A, your user
name may be User One/IT/Company A. In this example you could create an
organizational policies for */IT/Company A or */Company A.

59
An explicit policy is a policy that is explicitly applied to a user. This could mean that the
policy is defined in the users person document or is assigned to the user within the policy
document itself. When assigning a policy to a user or group within the policy document
itself, the policy is then considered to be a dynamic policy. For more information on
dynamic policies, you can refer to the Domino wiki article How Dynamic Policies can
reduce your administration workload. For more information on assigning policies, refer to
the Domino Administrator help topic Planning and assigning policies.

2.3.2. Using Policies to Standardize Secure and Simplify Your
Environment
Policies can be used to simply resolve a number of common problems.
Problem You Need to Solve The Policy Answer
Enforce corporate security standards such as
Changing passwords every x days
Password minimum length
Organizational security policy
Security risk from having Notes.id files stored on a
network drive with their default password
Organization security policy with an
assigned ID Vault for secure ID file
storage
Enable favorite calendaring features by default for
users for example:
Displaying new and unprocessed calendar and to
documents in the calendar
Automatically showing cancelled meetings as
cancelled in the calendar
Automatically checking for conflicts when adding
appointments to the calendar
Organizational mail policy
Configure server side archiving for all users Organizational archiving policy
Ensure all administrators use consistent settings
when registering new users
Organizational registration policy
60
Get the new sales application replicated locally to
your sales team.
A dynamic desktop policy assigned to the
sales team.
Configure default settings for all new Lotus Notes
Install
Organizational setup policy
Define default settings for mobile users A dynamic Traveler policy assigned to users
as they purchase a new mobile
device supported by Traveler
Automatically upgrade mail file design when a client
is upgraded.
Organizational desktop policy
Set a NOTES.INI parameter at the client for a
remote user
Explicit desktop policy

The list could go on and on. Policies can be a Domino administrators best friend or
biggest headache. Here are some definitions, hints and tips that will help you succeed
with your policy roll out:
Test first! Before rolling out a new organizational policy, create an explicit policy and
assign it to a small set of users. If it goes well, then create your organizational or
organizational unit policies for the rest of the employees.
Create policies based on your organizational structure. A policy implementation of
managers, employees and contractors will be too vague to work in most cases.
Assign policies via groups or individually within the policy document, policy
assignment tab rather than explicitly in the person document. Assigning policies in
the person document is time consuming and difficult to manage.
Create exception policies where needed for executives. Be sure to include a detailed
description to help you identify the policy. It is also recommended to hide the group
document from the general user population using a readers list. For more information
on hiding group documents, refer to Limiting access to group documents section of
the Mass mailings article.
When learning policies for the first time, the language used can be confusing. For
example, there is no option to create an organizational security policy. What does
that really mean? A policy of type organizational where the policy name matches your
organization, for example */organization, which has a security settings document
applied as shown in figure 3.
61

A desktop policy applies to all location documents. If you only want to apply a
specific setting to a single location you should still make this change manually.
There is no undo with policies. Once a policy has been applied, you can change the
policy to push out a new value, but there is no way to choose put it back to the
original value.
When using policies to push replica databases or bookmarks to clients, never delete
the database on the server without removing the database entry from the policy. As a
safety net, always create database redirects when you delete a database or template
to prevent a problem with a policy pointing to a non-existent database as shown in
Figure 4.
62

Know your environment. Policies will behave differently depending on the client
version being used. This would be expected as many new functions have become
available with the later client versions. Be sure your clients can support the function
you want to roll out.
A policy may be implemented by a number of processes. Archive policies are used
when running compact a or compact A. Mail policy values are rolled out by the
administration process (adminp) every 12 hours or immediately with the tell adminp
process mailpolicy command. A setup policy is read and used during initial client
setup. The remaining policy setting types are implemented by the Lotus Notes clients
dynamic client configuration (dcc). You can be certain dcc is running if you see the
entry Notes configuration settings have been refreshed in the status bar of the client.
Policies are stored within the users Contacts database (names.nsf). This database
should be using the latest design to prevent problems when using policies.
When using a desktop policy to control settings within iNotes you must also have a
mail policy. This is due to the fact that it is the job of the adminp task to update the
iNotes profile document from the desktop policy and adminp will only runs when that
process when a mail policy is present.
When using a How to apply value of Set initial value in a settings document, the
value you are setting will be pushed to the client when the policy is first saved and
any time the policy is modified or saved.
Policy signatures are important. Policies will only be applied and used when signed
by an administrator with proper authorities and a valid key. Thus, it is recommended
that you sign your policies with a generic administration account or resign policies
before the administration id can expire or when they administrator leaves the
company. To see who is the signer of a policy and settings documents using the
Policies and Settings views in the Domino Administrator client as seen in Figure 5.
By placing the policy or settings document in edit mode and saving it the signature
will be updated with your signature.
63

For a detailed description and an example of the different policy settings, refer to the
policies self training modules or review the Domino Policy Blog.

64
Part 3. Effective Server Administration

3.1. Monitoring
Monitoring a Domino environment means a repeating systematic collection and
supervision of the environment and its process and individual tasks within. The main
functionality of monitoring is to identify if certain parameters of a system or environment
exceed their defined boundaries and react in a defined way, for example, by alerting.

Due to the highly configurable nature of Domino and a variety of tasks it can perform, the
aim of this article is to define a monitoring strategy that covers the most common
components of a Domino environment. This monitoring strategy should be treated as a
base line that requires further customization to accommodate your specific Domino
environment.

3.1.1. Monitoring Options
We do not recommend a single solution or a single tool for all customers. Certainly large
implementations of Domino or those with high usage have different needs than smaller
Domino environments.

IBM offers the following monitoring options:
Server Monitor
This is a basic monitoring option built into the Lotus Domino Administrator client. It is
great for small environments or as an additional monitoring for one of the following
solutions.
Domino Domain Monitoring (DDM)
This is a server feature built into Lotus Domino and enabled by default. DDM is great
for detecting, understanding and acting on run time issues.
DDM probes log events. The administrators need to check the events that have been
logged. Event generators and event handlers together with statistics collection can be
used to monitor a Domino environment. For information on how they are handled,
see:
http://www.ibm.com/developerworks/lotus/tutorials/lsdom6stats/index.html
IBM Tivoli ITCAM for Applications
This is part of an enterprise class monitoring solution, which is extremely scalable
and capable to monitor much more than Lotus Domino alone.
Its functionality can be deployed agent-based or agentless. It leverages best-practice
models that focus on performance monitoring of key Lotus Domino components
including servers, mail routing, replication, calendar, database, and clusters. Deeper
Domino administrative capabilities are available with IntelliWatch
http://www-01.ibm.com/software/tivoli/products/composite-application-mgr-
applications/index.html
IBM Tivoli Intelliwatch Pinnacle for Distributed Systems
This is an automated problem detection and correction, system-wide product
configuration options, custom reporting capabilities, fault recovery and more. For
more information, see:
http://www-01.ibm.com/software/tivoli/products/intelliwatch/
65

Note, there are also 3rd party solutions on the market you can use which are not listed
here.
3.1.2. What should be monitored?
A common mistake is to limit monitoring to the application Lotus Domino itself. A number
of other components should also be monitored to ensure their work and to avoid
spending time on analyzing issues which are caused by a completely different area.

This list provides a brief overview of which elements should be monitored:
Network (LAN / WAN)
Platform (Hardware, Operating system)
Storage & Backup environment
Application with its components
Within this article, we focus on the last part Application which in our case is Lotus
Domino.
3.1.3. Monitoring Profiles for Domino
For ease of reference, different monitoring profiles should be defined within an
environment. By grouping monitors in this way, it is possible to create profiles which are
applicable to specific server configurations or functions in the Domino environment.

The following server roles shall be understood as an example:
Generic Domino Servers Applied for all domino servers.
Mail Servers Servers hosting end user mailboxes.
Web Servers Servers hosting Web sites (HTTP services).
Cluster Servers Servers providing cluster service.
Special Application Servers - Servers providing additional services.

Additional profiles can be defined based on your environment needs. Make sure to
document additional server profiles and include a definition when to use which monitoring
profile.
Action
Monitoring by itself is useless unless you take actions in case of an event or problem.
These actions can be defined for each response level and also for each event in detail.
Which action is the most important or convenient depends on your corporate
environment.

In small implementations of Lotus Domino, it might be enough to mail the administrator to
take action some time later. In large environments, there might need to have a solution
which supports 24x7 monitoring and alerting. In this scenario, it is often required to
integrate Lotus Domino monitoring results into an enterprise-wide monitoring system or
help desk system.

66
Actions depend to different factors like the size of the environment and the availability of
systems for alerting or ticket management.

Lotus Domino supports a number of notification actions which can be used further on to
build custom integrations to 3rd party systems, for example, to automatically open a help
desk ticket in your custom help desk application.

Figure 1 shows event handler methods.

If a Tivoli Enterprise Console is already available, then forwarding events to this console
is recommended. This is most likely the case for medium and large Lotus Domino
installations.

Hint: Cell Phone Alert
In order to receive cell phone alerts, there is a special cell phone configuration which
some providers support. Providers can be requested to forward email messages to a
phone as text messages (SMS). This allows notifying administrators via SMS when a
critical event occurs. If properly configured, a cell phone can receive email messages in
SMS form. The email must be addressed to a specific email address and domain name
which is defined by your provider.

In most cases, it is your cell phone number followed by the providers gateway domain
name. for example, 0123456789@.. Consult your cell phone carrier for details about how
to enable this configuration. Be aware that this may add extra cost to your phone bill.

To notify multiple people about the same event, create a group in your Domino directory
(e.g. AdminAlert-HTTPServers) which contains a list of these special email addresses.

Mapping Response Level to Severity Level
For further understanding the configuration details later on in this article, we map the
response level to severity levels which are widely used in help desk systems.
Response
Level
Severity
Level
Description
N/A Sev 1 Highest level of attention required, serious impact to
business expected.
67
Fatal Sev 2 High attention required, system is functioning but may
lead to service disruption if no action is taken
Critical Sev 3 Requires attention of a Domino administrator, if not
handled in a timely manner this may lead to further
problems
Warning Sev 4 Should be brought to administrators attention, but
doesnt require immediate attention
Reset N/A Previous severity now stabilized
Profile: Generic
A default monitoring profile should be applied to every Domino server, regardless of it is
designated role.

In general, where a monitor is considered important and critical enough that it will impact
server function, the monitor interval can be set to 5 or 10 minutes. Otherwise an interval
of hourly is predominant.

The Generic Domino Server Profile should include the following monitors:
Monitor Name Response
Level
Trigger Details and
Interval
Mail Probe Warning (high) On time out Mail Delivery
Monitoring
probe
Send Interval:
10 minutes
Time out
threshold: 10
minutes
Server availability Fatal (non-
clustered
servers)
Critical
(clustered
servers)
Reset
is
unavailable
is available
TCP Event
Monitor
Every 5 min
Task adminp Fatal
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative :
Hourly
Task event Fatal
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative :
Every 10
minutes
68
Task amgr Fatal
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative :
Every 10
minutes
Task stats Failure
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative :
Every 10
minutes
Task update Fatal
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative :
Every 10
minutes
Task router Fatal
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative :
Every 10
minutes
Task replica Fatal
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative :
Every 10
minutes
Task DAOSMgr Fatal Becomes
Unavailable
Task Status
Monitor
Alternative :
Every 10
minutes
Task MTC Fatal
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative :
Every 10
minutes
Domino Statistic
Replica.Failed
Warning
Failure
Increase of
10
Increase of
10
Statistic Event
Generator
Alternative :
Hourly
Domino Statistic
Server.Sessions.Dropp
ed
Warning
Failure
Increase of
50
Increase of
Statistic Event
Generator
Alternative :
69
100 Hourly
Domino Statistic
Server.Users
Warning
Failure
Increases
above X
Increases
above Y
Statistic Event
Generator
(X and Y
depend on size
of server)
Alternative :
Hourly
Domino Statistic
Agent.Hourly.Unsucces
sfulRuns
Warning Increase
Above 0
Statistic Event
Generator
Alternative:
Hourly
For details, see
IBM Technote
1232603
Domino Statistic
Agent.Daily.Unsuccessf
ulRuns
Warning Increase
Above 0
Statistic Event
Generator
Alternative:
Daily
For details, see
IBM Technote
1232603
ACL Change
(names.nsf)
Warning (high) On ACL
change.
Database Event
Generator
Monitor ACL
Change
File name:
names.nsf
Servers: all
Domino servers
in scope
ACL Change
(Admin4.nsf)
Warning (high) On ACL
change.
Database Event
Generator
Monitor ACL
Change
File name:
admin4.nsf
Servers: all
Domino servers
in scope

70
Profile: Mail Server
The following monitors will be applied to all Domino servers designated as mail servers:
Generic Monitoring Profile.

In addition, the following monitors are recommended:
Level
Trigger Details & Interval
Task calconn Fatal
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative : Every
10 minutes
Task sched Fatal
Reset
Becomes
Unavailable
Becomes
Available
Task Status
Monitor
Alternative : Every
10 minutes
Domino Statistic
Mail.Dead
Warning
Failure
Reset
Increases
above X
Increases
above Y
Decreases
below X
Statistic Event
Generator
Alternative : Hourly
X and Y depend on
size of environment
Domino Statistic
Mail.Waiting
Warning
Fatal
Increases
above X
Increases
above Y
Statistic Event
Generator
Alternative : Every
10 minutes
X and Y depend on
size of environment
Domino Statistic
Mail.Trans..Failures
Warning
Fatal
Reset
Increases
above 100
Increases
above 500
Decreases
below X
Statistic Event
Generator
Alternative : Hourly
X and Y depend on
size of environment

Profile: Web Server
The following monitors will be applied to all Domino servers designated as Web servers:
Domino Mail Server Monitors Profile (if needed).
71

In addition, the following monitor profile should be added:
Level
Trigger Details & Interval
Task http Fatal
Reset
Becomes
Unavaila
ble
Becomes
Available
Task Status Monitor
Alternative: Every 10
minutes
Domino Statistic
HTTP.PeakConnecti
ons
Warning
Failure
Increases
above X
Increases
above Y
Statistic Event Generator
(X and Y depend on size
of server)
minutes
For details see IBM
Technote 1232603
Domino Statistic
Domino.Threads.Act
ive.Peak
Warning
Failure
Increases
above X
Increases
above Y
Statistic Event Generator
(X and Y depend on size
of server)
minutes
For details see IBM
Technote 1232603

Profile: Domino Cluster
Any Domino Servers configured as a Domino cluster should have the following Domino
Cluster Server monitoring profile applied in addition to the basic profiles:
Domino Mail Server Monitors Profile (if needed)
In addition, the following monitor profile should be added:
Monitor Name Respo
nse
Level
Trigg
er
Details
&
Interva
l
Task clrepl Fatal
Reset
Beco
mes
Unav
ailabl
e
Beco
mes
Availa
ble
Task
Status
Monitor
Alternat
ive:
Every
10
minutes
72
Task cldbdir Fatal
Reset
Beco
mes
Unav
ailabl
e
Beco
mes
Availa
ble
Task
Status
Monitor
Alternat
ive:
Every
10
minutes
Domino Statistic Replica.Cluster.Failed Critical
Reset

1
<1
Statistic
Event
Genera
tor
Alternat
ive:
Every
60
minutes
For
details
see
IBM
Techno
te
123260
3
Domino Statistic
Server.Cluster.OpenRedirects.LoadBalanceB
yPath.Unsuccessful
Warnin
g
Critical
Fatal
Reset

1

5
10
<1
Statistic
Event
Genera
tor
Alternat
ive:
Every
60
minutes
For
details
see
IBM
Techno
te
123260
3
Domino Statistic
Server.Cluster.OpenRedirects.LoadBalance.
Unsuccessful
Warnin
g
Critical
Fatal
Reset

1

5

10
<1
Statistic
Event
Genera
tor
(X and
Y
depend
on size
of
73
server)
Alternat
ive:
Every
60
minutes
For
details
see
IBM
Techno
te
123260
3
Domino Statistic
Server.Cluster.OpenRedirects.FailoverByPat
h.Unsuccessful
Warnin
g
Critical
Fatal
Reset

1

5

10
<1
Statistic
Event
Genera
tor
Alternat
ive:
Every
60
minutes
For
details
see
IBM
Techno
te
123260
3
Domino Statistic
Server.Cluster.OpenRedirects.Failover.Unsu
ccessful
Warnin
g
Critical
Fatal
Reset

1

5

10
<1
Statistic
Event
Genera
tor
Alternat
ive:
Every
60
minutes
For
details
see
IBM
Techno
te
123260
3
Replica.Cluster.WorkQueueDepth.Avg Warnin
g

500
Statistic
Event
74
Genera
tor
Alternat
ive:
Every
60
minutes
For
details
see
IBM
Techno
te
123260
3
Example of documenting Monitoring Profiles
Server Name Generic Mail Hub Web Cluster Custom
App.
ServerA/ITSO Yes Yes Yes
ServerB/ITSO Yes Yes
SametimeA/ITSO Yes
3.1.4. Domino Event Monitoring
Although the profiles above can be implemented in different monitoring systems, it is
possible to monitor Lotus Domino event from the Domino Monitoring Configuration
(events4.nsf) database.

To prevent too much information from being shown, administrators should monitor all
Domino events defined as Fatal, Failure or Warning (high), as defined in the table below.
Each event type sub classifies each message with a severity level. These severity levels
are defined, in the Lotus Domino server, as:
Severity
Level
Response
Level
Meaning
Fatal Fatal Imminent system crash
Failure Critical Severe failure that does not cause a system crash.
Warning
(high)
Warning Loss of function requiring intervention.
Warning
(low)
N/A Performance degradation.
Normal N/A Status messages.
All N/A All of the above messages.
75
Severities

For best results you may wish to change the following default settings:
Remember to document changed defaults, so you can reapply them after an upgrade of
Lotus Domino to a higher version.
Value Text Old
event
severit
y
New
event
severit
y
Reason
0x02CC Database is
being
Compacted;
Compact
must finish
before use.
Warning
(Low)
Normal Compact task runs
against (e.g.) a
system database
which is in use.
0x0EA2 Recovery
Manager:
Assigning
new DBIID
for (need
new backup
for media
recovery).
Warning
(Low)
Normal Backup software is
requested to take
a new full backup
of this application.
0x0EA8 Recovery
Manager:
Restart
Recovery
complete. (/
databases
needed
full/partial
recovery)
Warning
(Low)
Normal This only indicates
that the server has
been restarted
completely.
0x0F13 Database is
currently
being
indexed by
another
process
Warning
(Low)
Normal This is only
informational.
0x0F3B Full Text
Error (FTG):
Exceeded
max
configured
index size
while
indexing
document NT
in database
index
Warning
(High)
Normal We do not want to
FT large
attachments - so
this error is
normal.
0x1104 Recipient
user name
not unique.
Several
matches
Failure Normal We cannot do
anything about,
because the
recipient is chosen
by the sender, and
76
found in
Domino
Directory.
when sent offline
or to email
address not
validated by Client.
0x1105 User not
listed in
Domino
Directory
Warning
(Low)
Normal Failure occurs
every time a user
writes wrong name
in SendTo field.
0x1149 Error
registering
mail rule for
database
Warning
(High)
Normal Rules is controlled
by users - we can
not fix this every
time - and it has
no consequence
for the server.
0x131B Database
created by
Warning
(Low)
Normal This is only
informational.
0x131C Database
deleted by
Warning
(Low)
Normal This is only
informational.
0x1321 ATTEMPT
TO ACCESS
SERVER by
was denied
Warning
(Low)
Normal Many users may
try to access
Admin server or
servers with
limited access,
e.g. because they
have had access
before.
0x1323 ATTEMPT
TO ACCESS
DATABASE
by was
denied
Warning
(High)
Normal Normal (ex. Users
try to see calendar
details and does
not have any
public access or
higher).
0x1357 Failing over
from for
replica id ,
directing
open to
Warning
(Low)
Normal Information about
an user has been
redirected to
cluster-server.
0x135C Failing over
from ,
directing
open to
Warning
(Low)
Normal Information about
an user has been
redirected to
cluster-server.
0x135E Unable to
redirect
failover from
Warning
(Low)
Normal Information that a
database was not
able to failover to
cluster-server
0x138C Operation
cannot be
performed at
the current
time -
database
compaction
Warning
(Low)
Normal Normal under
compact
77
in progress.
0x1519 A DDM
report
document
(NoteID 0x)
could not be
opened.
Warning
(High)
Normal If a DDM report
has been manually
deleted, and then
another instance
of the error is
logged, then this
error is coming.
0x1614 Replicator
was unable
to initialize
(from ):
Failure Normal Failure occurs
every time a
replica stub is
made.
0x19FC Your account
is locked out;
see your
system
administrator
to reset it
Warning
(Low)
Normal Many users forget
to change their
password in time;
we consider this to
be fixed by the
user himself.
0x330A documents (
bytes)
indexed in
Warning
(Low)
Normal Indexing is normal.
0x9AC0 LDAP
Server:
Warning:
Invalid
credentials
specified on
Bind request,
DN is
Warning
(Low)
Normal Normal behavior,
see IBM Technote
1219847.
0x331A Database
was marked
for delete
and has
been deleted
Warning
(Low)
Normal This is only
informational.
0x3320 Admin
Process:
does not
appear in the
ACLs of any
databases
designating
as their
Administratio
n Server
Warning
(Low)
Normal AdminP process is
normal.
0x3327 does not
appear in the
Readers or
Authors
fields of any
databases
designating
as their
Administratio
n Server
Warning
(Low)
normal.
78
0x3346 The
database is
transactionall
y logged. A
full backup of
it needs to be
performed on
for media
recovery.
Failure Normal Backup software is
requested to take
a new full backup
of this application.
0x336D Router:
Message
contains no
recipients
Warning
(High
Normal Information on
missing recipients
in a message.
0x33C4 does not
appear in the
unread lists
of the
databases on
.
Warning
(Low)
normal.
0x33E3 Admin
Process:
does not
appear in
design
elements of
any
databases
designating
as their
Administratio
n Server
Warning
(High)
normal.
0x3032 Not all
specified
languages
were found in
design
template
Normal Warning
(High)
This error has to
be handled,
otherwise refresh
design of the
database fails.
3.1.5. Further Reading
For more information about how to use and configure DDM, refer to the following IBM
Technote and the IBM Redpaper:
http://www.redbooks.ibm.com/abstracts/redp4089.html

79
3.2. Mail Routing
If your company is like most companies using Lotus Domino, you are using your server to
route mail. This article provides a brief introduction to Domino mail routing concepts and
helps you to learn more about how to manage a Domino mail environment. Since there
are many great resources already available, this article helps guide you to these
resources.

For a description of the tasks, databases and templates involved in mail routing, refer to
Mail routing with the Domino mail server.
3.2.1. Managing Spam
As a Domino administrator, one of the most common concerns is how to deal with spam.
There are a number of resources and options available. To understand how to use the
options available within the Domino server configuration, refer to the articles Controlling
spam: Advanced SMTP settings in Lotus Domino and Understanding SMTP
authentication and securing your IBM Lotus Domino 8 server from spam.

In addition to the SMTP settings, you may also want to create a server-based mail rule to
filter mail. For information on creating mail rules refer to the Domino Infocenter topic
Filtering new mail using rules.

In additional to the security capabilities in Domino, many companies use a spam filtering
service or software. One example would be Lotus Protector for Mail Security. If you are
using a 3rd party spam filtering service, you may want to configure your server that only
mail which is being sent from your vendor will be accepted by your server. You can easily
do this by defining SMTP Inbound Connection Controls. In order to do this, you will need
to obtain a list of hosts, IP addresses or IP ranges used by your vendor. You can then
define those entries as well as your internal hosts in the field Allow connections
only from the following SMTP internet hostnames/IP addresses. You
will find this setting in the configuration document on the Router/SMTP Restrictions
and Controls SMTP Inbound Controls tab. In the example shown in figure 1, you can
see an IP range, IP address, generic host name and the internal private network range
allowed for inbound SMTP mail connections. The host name must only match the end of
the name so in this case both server1.YourVendorDomain.com and
server1.mail.YourVendorDomain.com would be allowed to connect to the Domino server.
It is important to consider internal inbound connections if you have any other applications
or hardware such as a printer or copy machine that may route mail through your Domino
server.

80

In some companies checking outgoing messages is equally important as checking
inbound. If you are in the position that you want all outbound mail to be scanned for
possible spam by your vendor, you can easily do this by configuring your vendor as an
outbound relay server. You will find the Relay host for messages leaving the
local internet domain field in the configuration document for your server,
Router/SMTP Basics tab. In figure 2 the relay host is set to
Server.YourVendorDomain.com. Note that if you specify an IP address, it must be
enclosed in square brackets. Also, only one value is allowed in this field, so use caution
when configuring a relay server as a relay server failure will prevent all outbound mail
from routing.

3.2.2. Mail routing and multiple directories
When working specifically with mail routing, the topic of multiple directories is frequently
brought up. Maybe you need to have secondary directories with customer or vendor
contacts for mail addressing. Perhaps your users have been requesting a way to perform
address look-ups when disconnected from the server. Possibly you need to prevent the
server from searching secondary directories for mail addresses. Domino provides ways
to accomplish all of those goals using directory assistance, extended directory catalogs
and/or mobile directory catalogs. For more information refer to 3.4 Multiple Directories.
81
3.2.3. Journaling
Mail journaling allows administrators to keep a copy of specific messages or all
messages as they route through the Domino server. This can be important for security or
required for companies with pending litigation. Mail journaling is configured with a
combination of settings specified in the configuration document and a mail rule. To
access mail journaling settings open the configuration document and access the
Router/SMTP Advanced Journaling tab as shown in figure 3.

Some of the fields are rather self-explanatory while other settings will determine access
and usability of the journal. There are two methods available for journaling:
Copy to local database
Send to mail-in database

The default method is to Copy to local database. When this option is selected, data will
be automatically encrypted for the user selected in the Encrypt on behalf of user
field except for those fields listed in the Field encryption exclusion list field.

The second method is to choose Send to mail-in database. Why might you choose one
versus the other? The advantage of using a mail-in database is that multiple servers can
journal to the same database. The disadvantage is that as administrator you must
manage the encryption and database rollover as this will no longer be done for you.

Unless you have a tool to manage the mail-in database, using the default option of Copy
to local database is recommended.

The database management options are rather straight forward. You can choose to create
a new journal based on size or date. By default a new journal will be created each day. At
approximately midnight the current mail journal will be renamed to mjmmddyyyy.nsf, for
example mj11302010.nsf. The last field in the Basics section of the journaling tab is
Journal Recipients. Whether or not you enable this setting you will be able to see the
original values chosen in the TO, CC and BCC fields for the message. In some cases,
82
this may be a group. By default, you will just see the group name listed in the journal.
Who were members of the group? This could change. To ensure you see all of the actual
recipients of the message, you will want to set Journal Recipients to Enable.

Assuming you have chosen to copy the journaled message to a local database that the
server will manage for you, determining if the field encryption exclusion list should be
modified and which user should be used for encryption is slightly more complicated. The
values you choose will effect what data will be seen by users accessing the journal when
not using the ID listed in the Encrypt on behalf of user field and what data can be
included in a full text index. A full text index is built by the Domino server, so only data
that can be read by the Domino server can be included in the full text index. The following
table will help you match your company requirements with the proper journaling
configuration.
Requirements: Configuration Details:
Data must be
secured with
full data
access
restricted to
one or two
users.
Message
subject, body
and
recipients
must be
encrypted.

Full text
searching of
message
subject and
bodies is not
required.
This is the default configuration. To satisfy these requirements, register a new user and
specify that user in the field. For example, At fictional Software Company A , they created a
user named Mail Journaling/Administration/Company A. This user should then added to the
ACL of the mail journal and we suggest that the person document have a readers list to
hide this person from the general user population. The id file and password are then shared
and accessed only by the users designated at the company with this authority. Based on
the default field encryption exclusion list, anyone with reader authority to the journal will
only be able to read the date the message was sent and who was the original sender as
shown in Figure 4. When the message is opened with the Mail
Journaling/Administration/Company A id, the entire message is visible.

Depending on your security requirements you may want to further secure the id used by
creating the id in a private organizational unit and only provide password reset authority to
those administrators who are authorized to access the mail journal. For more information on
using an ID vault refer to ID Vault .
All users with
reader
access or
above to the
mail journal
application
should be
able to view
all messages
in the journal.
Users who
access the
journal must
In order to satisfy these requirements, the Encrypt on behalf of user field must
be blank which will disable encryption. The entire message can now by seen and data
access is controlled by the ACL of the mail journal as shown in Figure 5.

83
be able to
perform
complete full
text searches
of subject
and body of
all e-mails.

All user with
reader
access or
above to the
mail journal
application
should be
able to view
the date,
sender,
recipients
and subject
of all
messages in
the journal.
The
message
body should
remain
hidden
unless
accessed by
the
appropriate
id file.

A full text
search must
be able to
used to
search for
recipients,
senders and
subjects.
In order to satisfy these requirements, register a new user and specify that user in the field.
For example, At fictional Software Company A, they created a user named Mail
Journaling/Administration/Company A. This user should then added to the ACL of the mail
journal and we suggest that the person document have a readers list to hide this person
from the general user population. The id file and password are then shared and accessed
only by the users designated at the company with this authority. The field encryption
exclusion list should be modified to include the SendTo, CopyTo, BlindCopyTo and Subject
fields. Once done, anyone with reader authority to the journal will only be able to read those
new fields as shown in figure 6. When the message is opened with the Mail
Journaling/Administration/Company A id, the entire message is visible as shown in figure 6
.

Mail
journaling
has been
running with
encryption. A
new
requirement
or lawsuit
requires that
certain mail
journals be
sent for
An agent can be used to decrypt the documents in the journal if needed. For details
refer to technote 1089495.
84
review. Data
encryption
must now be
removed
from some
journals.

Journaling notes.ini parameters:
JournalLimitForwardToAdmins
JournalLimitForwardToMailAddress
JournalLimitForwardToDomain
JournalLimitForwardToSendCopyTo

3.2.4. Out of Office Notification
Starting with Domino version 8.0, there are two methods of out of office notification within
Domino. The first is controlled by an agent and the second it controlled by the router task
and has many benefits over the agent. For information on this topic refer to The IBM
Lotus Notes and Domino Out of Office service: Best practices.

3.2.5. Mail routing in a clustered environment
In a clustered environment, when the primary server is down you mostly likely want any
mail destined for users on the downed server to be delivered on its cluster mate. There
are several configuration pieces that must be in place in order for this to work. First, a
replica of each mail file must exist on another server in the cluster.

Second, cluster mail failover must be enabled in the configuration document for all
servers in the cluster as well as any Domino server that will be attempting to send mail
directly to the server. You can find the cluster failover setting in the Router/SMTP
Advanced Controls tab. The Cluster failover setting should be set to Enabled for last
hop only as shown in figure 7. The setting of Enabled for all transfers in this domain may
also be used; however use caution to avoid mail routing loops with this setting. What is
the difference? When failover is allowed for all transfers if a hub server is down, mail can
be redirected to another hub server. This setting should only be used in enterprise size
deployments and as the administrator you should ensure that both servers are able to
send the mail to the desired destination to avoid problems with the mail getting stuck on
an alternate hub.

85

The cluster failover parameter was configured via a notes.ini parameter in order releases
of Domino. If you have inherited an environment you should ensure that
MailClusterFailover is not specified in your notes.ini or is set to 1 to prevent problems
with mismatched settings. To see if the setting is currently in use on your server you can
use the console command show config mail* and review the output. MailClusterFailover
should not be included in the output just like the example shown in figure 8. If so, you
know that the notes.ini setting is being defined manually in the notes.ini or in a
configuration document and should be removed.

> show config mail*

MAILSERVER=CN=server1/O=Company A
MAILTYPE=0

Figure 8: Example of show config mail*

The next piece that must be in place for cluster failover to work is that the cluster.ncf
must be populated. The cluster.ncf file is a text file with a list of all known clusters and
cluster members. It is populated automatically when it connects with a server that is a
member of a cluster. In enterprise size environments the default cache size may be too
small and prevent failover, For more information refer to technote 1102957.

The final configuration piece that is required in order for mail cluster failover to work is a
fully populated cluster database directory (cldbdir.nsf). The cluster database directory
contains a list of all replica databases in the cluster. As an administrator, you can choose
to disable cluster replication for a database. If a database has been marked out of service
in the cluster database directory, then cluster failover will not occur. In figure 9 below you
can see the mail\utwo.nsf has been disabled on server1.

Lastly, if you cannot determine why the router task is not properly delivery mail to a
cluster server, you can enable additional debug logging by setting
RouterDebugClusterFailover=1. This setting is dynamic and can be enabled or disabled
86
using the set config command, for example set config routerdebugclusterfailover=1.
For example, here is an example of cluster failover working normally:
Error connecting to server Server2/Company A: The server is not responding. The
server may be down or you may be experiencing network problems. Contact your
system administrator if this problem persists.
Router: Cluster failover starting for server server2/Company A in domain COMPANY
A; mail file mail\uthree
Router: Cluster failover found server server2/Company A in cluster MailCluster
Router: Cluster failover found server SERVER1/COMPANY A is a cluster mate of
server server2/Company A in cluster MailCluster
Router: Cluster failover found local failover replica mail\uthree.nsf
Router: Failing over mail transfer from server2/Company A to [$LocalDelivery]; mail
file mail\uthree.nsf
Router: Message 005758B8 delivered to User Three/Company A

When no replica exists or the replica has been marked out of service then the following
message will be posted:
Router: Cluster failover could not find server by Rep ID for Server2/Company A
mail/uthree; Unable to find path to server. Check that your network connection is
working. If you have a working connection, go to Preferences - Notes Ports and click
Trace to debug

3.3. Mass Mailings
Mass mailings are mail messages addressed to a large number of users. It could be your
quarterly employee newsletter sent out to all employees or a sales flyer to all customers.
In any case, it is a message with a large recipient list that can affect normal mail delivery
on your server.
3.3.1. The Mass Mailing Problem
When the Domino server routes mail, for the most part, it works in a first in first out
basis. It is also a multi-threaded process that is able to send multiple mail messages at
any given time. When a large mail message is received, the router task of the Domino
server will use as many threads as it can to deliver the message. This can be a good
thing in that Domino will get the message sent to all of the recipients as quickly as
possible; but in the process, it may appear as the router is hung. In this case, you see
the router task taking CPU and memory, and all new mail will stay in the mail.box file(s)
while the router completes its work on the message. If the message has a large number
of recipients, users may see a slow down when accessing the server. This happens while
the router is parsing the list of recipients and determining which messages should be
delivered locally. This step is commonly referred to as the name lookup step and
involves briefly locking the $users view in the Domino directory (names.nsf). Since the
$users view is also required for user authentication to the server, these lookups can
cause a brief delay for the end user.
87
3.3.2. The Mass Mailing Solution
Optimizing the performance of mass mailings on your server can be achieved in a
number of ways. That is because there are many factors that determine the impact of
sending the message. This includes the group size, how the recipients are specified, how
the message is sent, who can access the group and router configuration.
Using Sub Groups
One simple way is to break up very large group into multiple subgroubs. For example, at
Fictional Software Company A, they send monthly technical tips to all of their customers.
They have created a group called Product B Customers. In this group they have multiple
subgroups listed Product B Customers A-K, Product B Customers L-Q and Product B
Customers R-Z. Sending a message to the group containing multiple subgroups will be
more efficient than sending a message to one large group. This is because of the way the
router must parse each message as it prepares to send the message to each user. Even
better would be to send a message to each subgroup, but most users will reject this idea
as it requires more work on their part.
Properly Using Mail Addressing
Are your mass mailings sent to a group listed in the to, cc or bcc field? The router task
must process a message differently if a large group is specified in the to or cc field versus
the bcc field. When processing a bcc list, the router makes a unique copy of the message
for each recipient. That is very intensive work. You can reduce this overhead by using the
to or cc field when appropriate or set Disable_BCC_group_expansion=1. By setting
Disable_BCC_group_expansion, you are preventing this unique copy process from
occurring. However, as a result, the users will no longer see their name or e-mail address
listed in the to, cc, or bcc field when they receive the message. For additional information
on the Disable_BCC_group_expansion notes.ini parameter refer to technote 1089346.
Using Low Priority
Another option when sending mass mailings is to avoid using your production mail server
or avoid sending mass mailings during primary business hours. If you use a specific
account for your large distributions, selecting your administration server as the mail
server for that account would be a simple solution to avoid causing problems for your
users. However; if you only have one server or many users that must send mass
mailings, then an alternate solution would be to force users to send mass mails as low
priority messages and therefore to be delivered off hours.

88
How do you set a message to be sent as low priority? This is done when composing the
message and selecting Delivery Options. From there the user can set the Delivery
priority to Low as shown in figure 1.

To force your mass mails to deliver during off-hours, you need to take two actions (1)
Define times for low priority mail routing and (2) configure a rule to only allow mass
mailings to be accepted for delivery if they are sent as low priority by the user. By default
the Low priority mail routing time range is from 12:00 AM 6:00 AM. You can change this
range to be the best times for your environment in the configuration document,
Router/SMTP Restrictions and Controls Transfer Controls tab as shown in
figure 2.

89

Be sure that if you take down your server each evening for backups that it does not
conflict with your routing time. You should also be sure that you have not disabled the
message priority functionality on your server. The Ignore message priority setting is found
in the configuration document, Router/SMTP Advanced Controls tab as seen in
figure 3.

90
The next step is to configure a server based mail rule to define what should be
considered a mass mailing and thus be sent as low priority. A server based mail rule will
affect all mail going in and out of the Domino server so be aware that if you set the value
to low you could reject inbound mail directed to a large number of users. In the example
seen in figure 4, any message addressed to more than 50 recipients that is not a low
priority message will be rejected. The user will receive a Delivery Failure Report stating
that the message was rejected for policy reasons.

Limiting Access to Group Documents

Limiting access to group documents can also be an effective way of limiting which users
in your organization can send mass mailings. A readers list can be placed on any
document stored in a Domino database, including group documents. To create a readers
list on a document, right click on the document and select Document Properties. On the
security tab, remove the check next to All readers and above. Now only the users or
groups with a check next to their name in the list as shown in figure 5 will be able to
access the document. It is critical that when you use a readers list that you check the
LocalDomainServers group in order to allow the group and group changes to replicate
throughout your Domain. It is also recommended to also include the LocalDomainAdmins
group. There is no undo when it comes to a readers list. If you lock the document down
and the only person able to access the document leaves the company, there will be no
91
way to unlock or see the document. Thus, it is important to protect yourself by selecting
the LocalDomainAdmins or other group as appropriate in the readers list.

In the case of a group document, if a user is not part of the readers list, then the user will
not be allowed to send a message to the group from the Lotus Notes or iNotes clients. If
an unauthorized user attempts to send a message using a POP or IMAP client, they will
receive a non-delivery report with the error Not authorized to send mail to this user or
group.
Server Configuration
There are a number of NOTES.INI parameters that can be used to further tune your
Domino server and prevent server problems with mass mailings. This includes
RouterMaxConcurrentDeliverySize, RouterMaxEffectiveSize and
RouterMaxEffectiveSizeIncAttach.

RouterMaxConcurrentDeliverySize allows the router to open only one copy of a
message at a time if it is greater than the specified size in bytes. This setting provides a
performance improvement for messages with large attachments sent to multiple users.
Users may notice a slight delay for a message with a large attachment to reach all
recipients, but message will be delivered. The proper size for this parameter varies
depending on your environment. Many customers find a starting value of 1048576 (1
megabyte) to be helpful.

RouterMaxEffectiveSize sends a delivery failure notification to a user if the message
they are trying to send is greater than the effective message size, specified in kilobytes.
The Effective size of the message is calculated by taking the size of the message times
the total number of recipients (effective size = message size in kilobytes * number of
recipients). Note that the message size used in this calculation does not include
attachments unless RouterMaxEffectiveSizeIncAttach=1 is set. How you set this value
depends on whether or not you want to include attachments in your calculations.
RouterMaxEffectiveSize is similar to the Maximum message size parameter that can be
configured in a configuration settings document (figure 6) or a rule configured based on
size (figure 7). The difference is that when using the Maximum message size or a
92
server based mail rule the recipient count is not considered.

93
Transferring the Work Out of the Router Task
One final possibility for dealing with mass mailings is to find a way to transfer the work
away from the router task and into another task. You could do this by developing an
agent to sent the distribution. The agent could break up the message sending to groups
of small users, queue the message for delivery at a later time or use any of the other
strategies discussed in this article. There are also a number of 3rd party tools that will
manage and create large mailings without impacting server performance.
3.3.3. Conclusion
In summary you have seen a number of ways you can manage mass mailings in your
Domino environment. Some options involve user training and others are transparent to
your users. Now that you understand the many options available to you, you can
determine which set of options are the right choice for your business.

3.4. Multiple Directories
As a Domino administrator, you may be asked to integrate multiple directories in your
Domino environment. This could be anything from creating a simple address book to
track customers or vendors, integrating two Domino environments or something as
complex as configuring single sign on (SSO) in your environment. For more information
regarding authentication options for Domino, refer to 1.4 Domino Authentication Options.
Alternately, you could be trying to find a way for users that exist in another directory to
access your server via the web (web authentication). This article provides an overview
and introduction to configuring secondary directories in your environment.

For a list of features available, refer to the help document Comparison of directory
catalogs and directory assistance.

If you are creating your first secondary directory, you should be aware that the directory
should be created with the Domino Directory (pubnames.ntf) template as personal
address books are not recommended for use with directory assistance or directory
catalogs. Once your directory has been created and populated, you can then begin to
configure access to the directory for your users. If you have multiple Domino servers in
your environment, each server that needs to access the directory should have a local
replica for use.
3.4.1. Condensed Directory Catalog, Extended Directory Catalog
or Directory Assistance
To determine whether or not you should be looking into directory assistance or directory
catalog and for an overview of the steps required to implement, refer to figure 1 below.
You can click on each entry in the diagram for information on accomplishing that step.
94

3.4.2. Hints and Tips
Some other things to keep in mind if you are creating your first secondary directory or
catalog:
Directories must be created using the Domino directory (pubnames.ntf) template.
Condensed directory catalogs should not be used on the server and cannot be used
for group authorizations. For example, a group listed in a condensed catalog cannot
be used to grant access to an application in an ACL, but an extended directory
catalog or a secondary address book referenced via directory assistance can be
used for group authorization.
95
To determine if any secondary directories are currently in use by your Domino server,
enter the Domino console command show xdir.
A server should always have access to a local replica copy of any secondary
directory. Use an * instead of the actual server name to have the server check for the
file name locally as shown in figure 2.

3.5. Server Clustering Options
This article addresses improving Domino uptime. Domino uptime can be improved in
many ways. One way is by making all components redundant and the second way is
making the infrastructuremore simple. Yes, simple. If you have a simple and clear
infrastructure, it runs better. In this article, we discuss Domino components only.
Redundant power, cooling, network, HDD and other components that lay below Domino
need to be redundant as well.
3.5.1. Keep It Simple
The rule of thumb is not to mix configurations. If you have a mail cluster, let it provide
mail to users. Do not put other things on this server, including all applications, web
server, and products, such as Lotus Quickr, Traveler, and Sametime. For these types of
products, you should use a separate server. On top of Domino, you can have only one
such add-on product. So even if you have an additional server for such products, do not
put them all on top of each other.

IBM Domino license states that if you need to install Traveler, you do not need to buy
additional licenses (for additional Domino server). An advantage of putting Traveler on a
separate server is that it improves security, because Traveler will be placed in the DMZ
zone. Even if somebody does hack it, it will be an empty server. Nothing else will be
compromised because all databases and applications will be located on other servers
that are not available from the internet. If you use Sametime Entry to enable users
chatting with each other, or you use Sametime for audio and video conferencing, then
again, you should place it on a separate server. This will make your servers and services
less dependent from each other.
Tips: Not every such product will work with any patch level of Domino. For example, a fix
for your mail environment, might be not compatible for the Microsoft Quickr server
96
running on the same machine. Before you upgrade to production environment, make a
clone and test to make sure it works fine first.
3.5.2. Redundant Domino Parts
There are different approaches on how to improve availability of the service to users and
systems that access Domino. Depending on how users and applications access Domino,
there are different ways on how to achieve this goal.

Is it possible to measure availability of a server? Yes. There are two factors in this
calculation: PLANNED downtime and UNPLANNED downtime. Planned downtime is
unavailability of a server which was forecasted and planned beforehand and it does not
impact users. For example, you plan to put a new version of Domino or patch operating
system during off hours. No one will notice that you brought the server down. After you
complete your task you bring the server back online. On the other hand, if there is a
power failure during the normal working hours or the operating system or Domino
crashes, this is considered UNPLANNED downtime. During this time, users cannot send
or receive mails or access databases. They will call the help desk and register support
calls. This unplanned downtime impacts the user's work activity.

The following table lists different approaches you can use to reduce unplanned Domino
downtime depending on the size of your Domino environment (small, medium, or large)
Approach Small Medium Large Notes
Domino Cluster + +++ +++++ Seamless failover
for Lotus Notes
users.
Note #1
Note #2
OS cluster +++ ++ + Note #3
Note #4
ICM-Internet
cluster manager
++ ++ +++
Hardware Load-
Balancer
+ ++ +++
POP3/IMAP/HTTP
proxy
+ + +

97
The following flowchart provides guidance on the approach best suites to your
environment to reduce system downtime.

Notes:
#1. Requires Enterprise or Utility license for each server.
#2. Requires Database adaptation for failover. All data (as determined by the Domino
Administrator) is duplicated on the servers in the cluster. If data is corrupted on one
server and deleted during a consistency check (Fixup), it will be automatically restored
(replicated) from other. If data is deleted on one server, it is deleted from the other
servers as well.
#3. You can use Domino Express licenses if your organization is less than 1000 users.
#4. Servers uses one disk to store data. So if one server goes down, the other server is
up using the same data. If data is corrupted, both servers will operate on corrupted data.
Servers use the same data.
We explain how you can reduce the downtime of your Domino server using various
approaches for the remaining sections of this article.
3.5.3. Domino Cluster
Small:
Medium:
Large:
Choose this approach if: Organization size is Medium, Large or Domino downtime in
unacceptable.
98

A Domino cluster is a group of servers of two or more servers that provides failover for
Domino applications. Failover is a process in which the Lotus Notes client is redirected
from one server to another when the primary server is not responding or is overloaded.
The advantage of failover is that users will continue to have access to critical resources.
In the most current versions of Domino, user will not receive an error messages and the
failover happens transparently for the user.

In case of a Domino cluster, database replicas of clustered databases are located on
different servers. Clustering provides not only failover, but also load balancing.
Overloaded servers can pass users to other servers that are not so busy.

In general, there are two types of operating system clusters, Active-Active, and Active-
Passive. For Active-Active cluster, all servers are available and serve users' requests at
the same time. For Active-Passive cluster, when one server (the Active server) serves
users' requests, the other server (the Passive server) waits and does not serve users.
When the Active server goes down, the Passive server notices that and it will become the
Active server and will start providing service to users.

Domino clustering requires a license for every server included in the cluster. It needs to
be purchased from IBM or an IBM Business Partner. Thus, there are additional license
costs for this solution. Some databases automatically support clustering and failover.
Other databases, developed in house, do not support clustering by default and need to
be programmed to support clustering.

Some databases provided by IBM already have clustering support, for example Mail
Database. Designer Help lists LotusScript functions that help to make databases work in
a cluster. For instance, Database.Open is a regular database open function and
Database.OpenWithFailover is a cluster version of it. Database.OpenWithFailover will try
to open database. If the database is not available, it will automatically failover to another
server. Enabling a database to support cluster mode is not about substituting one
function with another. There are different challenges that need to be solved by
developers. For instance, two documents are modified at the same time on different
99
servers. You need to think about document locking to prevent Save Conflicts. Agents are
another issue to consider in a clustered environment. Domino supports "After new mail
has arrived" agents failover, but it does not support scheduled agent failover.

The following technique can be used to solve scheduled agent to work in a clustered
environment:
There is one master agent that triggers all other agents (slave agents). This master agent
is scheduled on Any server. It is executed every X minutes on all nodes of cluster. To
avoid this agent to actually running on all servers at once, we need somehow try to run
the agent first on one server. If it is down, then we run the agent on other nodes. You can
create a profile document in a database where this Master Agent lives. You can put two
fields in this profile: PrimaryServer and TimeStamp. PrimaryServer is the name of server
on which agent should run and trigger all other agents. When the master agent runs
successfully on one server, it updates profile document with a timestamp and this profile
document is replicated to other servers via cluster replication. Master agents on other
servers checks if there is an up-to-date timestamp. If yes, then they quit execution as
they know that the Master agent on the primary server is already at work. If there is a big
gap, between NOW() and the last timestamp, then the Master Agent on the other server
understands that the Primary server is dow, and it backs up the Master agent. It then
triggers all of the other agents.

Nodes of a Domino cluster can be located in different buildings, cities or countries.
Domino provides the Geo-Clustering option. In case of fire, or if it is impossible to work in
the building, all information is available from the alternative location.
Summary
Domino Clustering is the best way to provide high-availability to Lotus Notes users.
Failover occurs and is seamless in the more recent versions of Domino, otherwise a
prompt to redirect to a different server is displayed. You may build a Domino cluster on
top of different operation systems. If you cannot provide cluster awareness of your
applications, you can choose other solutions such as an operating system cluster.
Additional Resources on this.
Understanding IBM Lotus Domino server clustering
http://www.ibm.com/developerworks/lotus/documentation/d-ls-dominoclustering/index.html
Lotus Domino R5 Clustering with IBM eServer xSeries and Netfinity Servers

3.5.4. OS Cluster
Small:
Medium:
Large:
100

One type of clusters is the operating system (OS) cluster. There are two or more servers
and Domino process is running on one server. When there is a need to switch server or
there is a hardware failure, the other server starts the Domino process on the other
server. In both cases, server will run under one and the same SERVER.ID and same IP
address. In case of Domino cluster, they have different names. If ServerA is running, and
we need to do some maintenance, we give command to the other server to take control.
Then the same ServerA is started on the other node. Users may experience a small
delay when the first server goes down and the second server starts up.

Almost every operating system provides an option to build an OS cluster. An additional
licence may be required for this. For example, on Linux -Heartbeat daemon provides a
solution to build cluster on Linux OS basis. In configuration of this daemon, you define
primary node and processes that need to be monitored. If one server goes down, the
second node takes control. It will map shared disk, where Domino data is located and will
start Domino server on another machine. The OS clustered Domino server appears to
end users under the same name and same IP address. If there are systems that access
Domino by host name, such as POP3/IMAP/LDAP/HTTP, they will successfully reach the
server.

If you have less than 1000 users in your company, you may use the Domino Express
licence for the OS cluster. Since a limitation of the Express licence is that you have less
than 1000 users and that Domino clustering is not used. From that point of view, you can
have Domino high availability at relatively low price.
101
If you use an OS cluster, then you will NOT use the Lotus Notes client failover feature,
from Release 8.x and 8.5.x. Domino supports failover for opened databases. in case of
OS cluster, users may need to re-open databases.

You MAY use an OS cluster in conjunction with Domino cluster. This is a supported
configuration.
3.5.5. Internet Cluster Manager (ICM)

In the earlier sections, we discussed high availability of the entire Domino server. Starting
from Release 5.x, there is an option for high availability Internet Cluster Manager, also
referred as ICM. ICM provides a failover for WEB clients who access your iNotes server
or intranet and company homepage. ICM is an additional task that is loaded on a Domino
server. It is quite easy to setup ICM if you already have a working Domino cluster. ICM is
an addition to Domino cluster, and it works only with Domino cluster.

In the Domino configuration you define which Domino cluster ICM should look for, if you
have many clusters in your Domino environment. ICM plays a re-director role, like a
dispatcher who (re)directs landing planes. When a new HTTP request comes in, ICM
knows which servers are available and redirects users to one of them. When one server
goes down, ICM notices that and the subsequent new requests are directed to another
available server. After some time laps, ICM sends Domino a ping command to check
which servers in the clusters are available. When new requests come in, ICM knows
which servers are up and which ones are down and sends the new requests to the
running servers.
102
A working scenario of ICM
We have two mail servers, mail1.company.com, and mail2.company.com. ICM listens for
requests on webmail.company.com. When users type the webmail.company.com
address, this request goes to ICM. ICM already knows which servers are up and it will
send the user redirection requests. The URL is changed to mail1.company.com or
mail2.company.com in users browser and the user is asked to authenticate.

It is advised to run ICM on a separate server than the Domino servers. Otherwise, if ICM
runs on an overloaded server, there is a probability that this server may become
unavailable and ICM will be not reachable for clients. If that occurs, ICM will not redirect
requests. If the ICM runs on a separate machine it just sends back redirect information to
clients. The ICM system should have low unplanned downtime.
What will happen if ICM is down
The cluster will be up, but users will not be redirected. If you want to improve availability
of ICM you can have several ICM servers and assign multiple IP addresses to one host.
Then, the failover of ICMs will happen at the DNS/OS level. With this approach you can
have a high level of ICM availability.

Deploy Single-Sign-On for web servers will give your users additional benefits. If mail
servers share one common LTPA Token, then if users change the URL from
mail1.company.com to mail2.company.com, there will be no additional login screens. If
one mail server goes down while the users read mail, users have to go back to
webmail.company.com host, and ICM will redirect users to another available server. With
Single-Sign-On set up, users will get to their mails without additional login forms.
3.5.6. iNotes High Availability Configuration
For information on iNotes high availability configuration, see
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/iNotes_High_Availability_Configuration
3.5.7. IMAP failover (Domino 8.5 new feature)
Release 8.5.2 of Domino added new functionality to IMAP users. Now IMAP users can
failover to different servers, and the IMAP client will not be confused. If you have many
IMAP users, you may benefit from this feature. There are some additional steps needed
to configure IMAP support on Domino Clustered servers so refer to Technote #1429885
or the 8.5.2 Administrator help. In conjunction with multiple IP addresses assigned to one
host, or software proxy, IMAP users will be redirected between available servers.
For additional information, see
Failover and the IMAP server
3.5.8. Lotus Traveler Server High Availability
At the moment Lotus Traveler does not provide an option for native clustering. An
enhancement for Lotus Traveler support is registered. You can still build many Traveler
servers. If a user manually changes the IP address of the server, the Traveler client will
do a Full Sync of Data, which is quite time/traffic consuming task. This is because of the
design of Traveler server.
103
One of the best solutions to make Traveler available in cluster is to put it on OS cluster or
use Proxy server which works in Active-Passive mode.
Related Topics:
Traveler Clustering and failover
http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.help.lnt8
51.doc/Clustering_and_failover.html
3.5.9. Load Balancer

One more option for improving availability of the servers is load balancer. Load balancer
is a hardware device or software that can check if target servers are available. When new
request comes in, it will redirect request to one of the available servers. Hardware
balanced can be used to cluster POP3/IMAP/SMTP users between servers.

In addition to POP3/IMAP/LDAP/SMTP protocol failover, you can use load balancer to
switch Lotus users between Domino servers. If you want to use IP-Sprayer (load
balancer) with Lotus Notes, you should have additional parameter described in Technote
1233210. You can deploy this parameter with the help of policy/desktop settings.
Notes client fail to connect to Domino servers behind a network sprayer
104
3.5.10. Software proxy (IBM HTTP, nGinx, etc)

There are software programs that work like proxy servers, and they can do automatic
failover of servers from which they request data. Some solutions like IBM HTTP can
provide failover (reverse proxy) of HTTP/HTTPS protocols. Some others can serve
SMTP, POP3, IMAP, HTTP for example NGINX. There are different vendors and every
proxy functionality is different from the protocol prospective. Depending on your need, if
only HTTP failover is needed or additional protocols like SMTP, POP3, you can select
which solution to be used.
3.5.11. Sametime and QuickR High Availability
Sametime and QuickR server may be also clustered. If these resources are critical for
you, deploy clustering which will provide high availability for Lotus Sametime or Lotus
QuickR users. Follow the links below to be guided how to deploy cluster for Lotus
Sametime and Lotus QuickR.
QuickPlace clustering guide: configuration and managing places
105
3.5.12. Disaster Recovery Plan
This section deals with Domino recovery if you have OS / hardware failure. Recovery
plan is a document that defines a sequence of actions and responsibilities during server
restore. Test recovery is a procedure that needs to be done after you deploy a new
backup solution. In addition to this, a test recovery needs to be repeated every year to be
sure changes done in the environment are reflected in the backup policy. Test recovery
shows that everything is fine with backup procedure. Test recovery should be ordered
without the backup team, so they will not prepare additional (full) backup copies. The
purpose of this to understand what problems you have in the test case, and eliminate
problems in future.

When you put a new server in production, be sure that this server is included in daily
backup procedure. Nobody knows when recovery will be needed or what data will be
needed. It can be one text or configuration file, stand-alone mail file, database that is part
of application or the entire server. It is vital to have a recovery plan for your Domino
environment. You should write it like you are going on vacation, you know that there will
be problems, and you do not want to have calls to your cell phone. Your colleagues
should be able to do the recovery according to your documentation. Describe what need
to be restored, how to restore them and in what sequence, installation locations, IP
addresses and phone numbers. This document should be kept up to date. It should be
printed and stored in an available place. Do not keep this only in electronic format. If the
system is down you will not be able to access it.

It is advised to do the test recovery of the entire server. You can do this on a separate
machine, a test server. Be sure to restore it on an IP isolated machine so when you bring
the server up it does not replicate other production servers. If you do the full recovery
once, you will be able to do this again smoother and faster in a real life. Do spend time
describing the steps you performed in the document. In a real life, you will do this at least
several times faster than the first time. Test the recovery. Find and highlight the things
you may have documented wrong in your current backup plan. For example, you backup
.nsf files by a Domino specific backup solution and backup everything else with an OS
backup solution, except the Domino DATA folder. In that case some important files, such
as cert.id, server.id, notes.ini may be excluded from backup. Test recovery is ensuring
that everything is fine with your backup solution and approach.

In your recovery plan describe sequence of the restore. How should the entire server be
restored, one mail file, or one document (alternative location, then copy paste).

3.6. Transaction Logging
Transaction logging is a real time log of transaction occurring on your Domino server. If
you are starting with transaction logging for the first time refer to the article Notes/Domino
Best Practices: Transaction Logging.
3.6.1. General Transaction Logging Recommendations for 8.5.x
Servers
When working with transaction logging at Domino version 8.5.x, your first decision must
be the type of transaction logging. If you are using a backup utility that will manage the
transaction logs for you or need point in time recovery, then you will want to use archived
106
style logging. Be aware that archived logs are not cleaned up by Domino automatically.
When using archived style logging, you must use a backup utility to clean up the old logs
to avoid filling all available disk with old logs. If you are not going to be a backup utility to
manage the archived logs or performing point in time recovery, then you should choose
to use the circular logging style. The maximum size for circular logs is 4GB and that is the
recommended value for all but the smallest implementations of Domino.
One source of confusion over many years is where to place the log directory. It may be
easier to state where you should not place your log directory. It should not be placed on
the same physical disk as your Domino data directory. That means you may place it
within the data directory when using a RAID disk array (like IBM i). For more information
refer to Transaction Logging Best Practices hardware recommendations.
3.6.2. Transaction Logging Tips
There are some things that every Domino administrator should know in order to prevent
problems with transaction logging:
Domino must always be able to obtain write access to the transaction log files, thus
you should not run anti-virus on the transaction log directory.
The transaction log tracks database changes through the DBIID. Thus, you should
never have multiple .nsf files with the same DBIID. The Lotus Notes client and
Domino server will ensure this never happens. However; if you are moving, when
copying files or restoring files at the OS level, be sure that you do not accidentally
duplicate a DBIID. If you need to make copy or restore a file at the OS level, you can
easily protect yourself by running a fixup or copy-style compact on the current
database to change the DBIID before you make the copy or restore.
If you have a problem with the transaction logs, always keep a copy of the log files,
console.log and any logasio*.txt files you may have in the
IBM_TECHNICAL_SUPPORT directory. Without that information, Lotus Support may
not be able to assist you in determining the cause of your problem.
To determine if a system database should be have transaction logging enabled, refer
to the table in the DAOS Best Practices article.

3.6.3. NOTES.INI Recommendations for Domino 8.5.x Servers
To help optimize your transaction logging configuration, there are some NOTES.INI
parameters that you should be aware of and may want to set on your Domino server.
NSF_DONT_LOG_MAILBOX_BODY=1: This setting will prevent the message body from
being logged for messages in the mail.box file(s). Since the mail.box files are very busy,
this can improve the performance and throughput of your mail.box file(s). If your mail files
are transaction logged, the message will then be logged at the point the message is
delivered rather than at the mail.box files.
RM_NO_LOG_LARGE_OBJECTS=1: No attachments larger than 1 MB placed into the
server mail.box file(s) will be transaction logged. This can improve the efficiency of mail
delivery for large messages.
107
Schedule_DisableTXNLogging=1: This setting will disable transaction logging for the
scheduling databases (busytime.nsf or clubusy.nsf).
3.6.4. Domino 8.5.x and ODS 51 Updates
Starting at Domino 8.5 and ODS level 51, log compression is not enabled. Transaction
Logging will store the data in the same format as the database. If the data is already
compressed, it will be compressed in the transaction logs. If the data is not compressed,
the data will not be compressed in the transaction logs. What data is being referred to
here? It could be data documents, design documents or attachments.

There are some cases where, for performance reasons (most often on system z and IBM
i systems), you may want to compress your transaction logs, but not compress your data.
In this case there are two NOTES.INI settings that can accomplish this for you:
NSF_COMPRESS_TXN_LOGS - Compress both design and data documents within
the transaction logs.
NSF_ENABLE_LZ1_TXN_LOGS Compress attachments stored within the
transaction logs using the LZ1 algorithm whether or not they have been stored
compressed within the Domino database.
Enabling LZ1 compression for attachments
To enable LZ1 compression in your environment, open the Lotus Domino Administrator
client, connect to your server, and select the Files tab. On the right panel, select
Advanced Properties. Select the option to enable Use LZ1 Compression and select
OK to apply as shown in figure 1 below.
After this option is set, the LZ1 compression will work from this period on all new
attachments, everything before this activation will remain the same way (without
compression). To covert the existing attachments to LZ1 compression, you can run the
compact task with the -ZU parameters.
Enabling design document compression
Open the Lotus Domino Administrator client, connect to your server, and select the Files
tab. On the right panel, select Advanced Properties. Select the option to enable
Compress database design and select OK to apply as shown in figure 1 below.. In
order to compress the design,a copy-style compact must now run on the database.
Enabling document data compression
Open the Lotus Domino Administrator client, connect to your server, and select the Files
tab. On the right panel, select Advanced Properties. Select the option to enable
Compress document data and select OK to apply as shown in figure 1 below.. In order
to compress the existing documents a copy-style compact must run on the database.
108

3.7. Domino Attachment Object Service (DAOS)
The Domino Attachment and Object Service (DAOS) is a new feature introduced in
Domino 8.5. DAOS is a mechanism which shares identical attachments between
databases on the same Domino server. When configured, separate and complete
attachments are no longer stored within a database, rather a single copy of an
attachment is stored on the file system with documents referring to them on the disk.
DAOS is implemented such that the creation and retrieval of attachments is transparent
to all transactions, including all user functions. The benefit of DAOS is file size reduction.
Any Domino database is eligible for participation in DAOS.

This article discusses when to deploy it in your environment.
For information on DAOS architecture, see:
IBM Lotus Domino going green: The new Lotus Domino attachment and object service
109
The following flowchart provides the guideline on when to deploy DAOS.

There are various technotes and articles available about DAOS including how to
configure it and how to estimate its impact.

How can you be sure if every server needs this? Should it be enabled on Traveler? No!
Should it be enabled on company cluster servers hosting several hundreds of gigabytes
of data? Definitely Yes!

When DAOS is used on a server, additional tasks are running on a server. DAOS also
requires changes in your backup procedure; because in addition to NSF files the .NLO
files need to be backed up.

110
DAOS is introduced in 8.5.0, but to be on a safe side, before implementing DAOS, you
should make sure you are on the recommended version of Domino for optimal DAOS
operation.
You also need to disable "Shared Mail" if it is used. Use this above flowchart as a
guideline. Do not run DAOS Estimator during working hours. Do this after working hours.

You can also use DDM(Domino Domain Monitoring) to monitor DAOS.

The following references illustrate the configuration and administration requirements for
DAOS:
Download the Domino Attachment and Object Service Estimator Tool version 1.5
IBM recommendation for conversion to DAOS enable a database
IBM Lotus Domino going green: The new Lotus Domino attachment and object service
Achieving ultimate storage and server cost savings with DAOS in IBM Lotus Notes and
Lotus Domino 8.5
IBM recommendation for conversion to DAOS enable a database
DAOS Backup and restore

3.8. Managing Domino Indexing
Without an index, finding the data you are looking for would be extremely difficult. There
are 3 different types of indexes used when using Lotus Domino -- view indexes, database
full text indexes, and Domain indexes. There are also multiple tasks involved with
Domino indexing including update, updall, chronos, kvoop and domidx. If you are new to
Domino administration, this can be overwhelming. This article gives you the information
you need to get started with and manage Domino indexing.
3.8.1. View Indexes
What is a view index? Anytime you open a Domino database, you are seeing information
presented based on a view index. You could be looking at the Inbox of a mail file, a
personal folder or a view in a custom application, but how that data is presented to you
and whether or not you are seeing the latest update depends on the view index. Since
view indexes are stored within a Domino database, you may wonder how to see if a view
index exists, if it is up to date or the size of the index. The Domino Administrator provides
a tool to easily see this information. To access the tool you will go to the Files tab in the
Domino Administrator client and right-click on the database you want to see. You should
then select Manage Views as seen in figure 1.

111

Within the Manage the views of this database window, you can see the name of each
view index, the index size, the creator of the view, refresh interval, discard interval and
the internal note id representing the view. In figure 2 you can see the views for a typical
mail file.

Now that you understand how view indexes are used and where they are stored, you
may want more information on how the view indexes are updated and the tasks
responsible. For this information refer to the Domino administration help topics Indexer
tasks: Update and Updall and Keyboard shortcuts that update or rebuild views.

3.8.2. Full Text Indexes
A full text index is an index designed to be used for searching within a Domino database.
The full text index may only contain words found within a document or may optionally
include text within numerous supported attachment types. A full text index is created and
stored outside of the application in a .ft folder. A user can easily see if their mail database
has a full text index by the icon displayed above the search bar as shown in figure 3.

112

When creating a full text index, there are several choices that need to be made including
the update frequency and whether or not attachments should be read and included in the
index. Be default, attachment indexing is turned off, but the default update frequency is
different depending on the method used to create the index as shown in figure 4. If a user
selects to create the index from within their mail file by clicking Not indexed as seen in
figure 3 the index will be created with an immediate update frequency. This is typically
not recommended for most applications. When creating the index from the Domino
Administrator, the default is daily updates, which is recommended for most applications.

You will also notice that the wording for the attachment indexing is also different. The
most complete attachment indexing is binary indexing performed by the kvoop task which
implements KeyView technology for reading the attachments. This type of indexing is
referred to as with file filter in the Domino Administrator client and called conversion filters
on the database properties Create Full-Text Index panel.

There are additional options which can include indexing encrypted fields, sentence and
paragraph breaks and case-sensitive searches. In general it is best to create the index
with the fewest index options needed based on the application to minimize the size of the
full text index.

Having a full text index on a database has some pros and some cons. Creating and
maintaining a full text index requires additional resources on your system. However, if a
database is not full text indexed and a user needs to perform a search, a temporary index
is created and then destroyed. If this is done many times, it would be more efficient to
have a permanent full text index. If an agent performs full text searches and a full text
index is not created for that database, the server will advise you to create a full text index
by logging the message Warning: Agent is performing full text operations on database
'mail/database.NSF' which is not full text indexed. This is extremely inefficient.

Because of the cost involved with building and maintaining temporary and full text
indexes, there are numerous options available to Domino Administrators to control who
can create an index, how often and what can be full text indexed. See the table below for
the options.
113
Scenario Details Pros/Cons
All users allowed
to create full text
indexes at their
discretion with
any refresh
interval they
choose.
In order to create an index, the user
must have a minimum of designer
access to the database. The user can
create the index within the Application
properties, the Full Text tab.
Pro: Does not require
administrator
intervention

Con: This is not best
practice. IBM
recommends a
maximum access
setting of editor in the
ACL of the mail file.

Con: Users can create
indexes with an
automatic or
immediate update
frequency on large mail
files which can impact
the performance of the
entire server.
No full text
indexing is
allowed except if
created by an
administrator
using the
Domino
Administrator
client.
As an administrator you can prevent
users from creating their own full text
indexes by setting
UPDATE_NO_FULLTEXT=1 into the
NOTES.INI on your server. With this
setting the following will occur:
Current full text indexes will continue to
be updated.
Administrators can create new full text
indexes from the Domino Administrator
client.
Users cannot create a full text index.
Pro: Administrators can
fully control when a full
text index is created
and the update
frequency used.

Pro: Prevent
performance problems
created by having mail
files with full text
indexes set for
automatic or
immediate updates.

Con: Users may be
upset if they have been
able to create full text
indexes in the past.
Available disk
space on the
system is low
and you want to
conserve space
by removing
unused indexes.
By default, view indexes are purged if
they have not been accessed in 45 days.
You can reduce this by
settingDEFAULT_INDEX_LIFETIME_DA
YS=<# of days> in the NOTES.INI of
your server.
Pro: Disk space used
by indexes that are not
being used can be
reclaimed and reused.
Con: If you set the
value too low, views
may need to be rebuilt
more frequently than
before and thus it will
have a negative
performance impact.
No temporary
indexes are
allowed on the
server. If a
database/applica
tion must be
searched, then a
full text index
must be created.
As an administrator, you can prevent all
temporary indexes from being created
and deleted by setting
FT_FLY_INDEX_OFF=1
Pro: Conserve server
resources by
maintaining full text
indexes rather than
creating and deleting a
temporary index each
time the user performs
a full text search.

114
Con: If you do need to
search a database only
one time, you will need
to manually create and
delete the full text
index.
No temporary
indexes are
allowed to be
created on the
specific
application.
At ODS version 48 or higher you can set
a database property called Dont allow
simple search. When this property is
selected, the database does not contain
a full text index and a user attempts a full
text search a message stating
Application must be full text indexed
before search is allowed.
Pro: You can avoid the
server overhead of
creating a temporary
full text index, but not
prevent it for all
databases.

Con: May lead to help
desk calls for users that
want to be able to
perform full text
searches.
It is desired to
force or prevent
the full text
indexing of
attachments or
attachment
types.
There are several ways that an
administrator can control attachment full
text indexing.

To disable all attachment indexing set
FT_Index_attachments=2

To force all full text indexes to include
attachments set
FT_Index_attachments=1

To exclude specific attachments types
from being indexed set
FT_INDEX_IGNORE_ATTCHMENT_TY
PES=<list of file types separated by
commas

Pro: As an
administrator you can
set a standard for all
attachments on the
server.

Pro: You can easily
disable attachment
indexing which will lead
to smaller full text index
sizes as well as fewer
system resources
needed when building
and maintaining
indexes.

Con: Unable to
accommodate
situations where
applications have
different requirements
which determine
whether or not
attachment indexing
should be used.
Need to disable
binary
attachment
indexing (kvoop
process).
As an administrator you can prevent the
binary attachment indexing process, also
known as the keyview filter, from
indexing attachments by setting
FT_BINARY_FILTER_OFF=1
Pro: You can easily
prevent the kvoop
process from running
on your server.

Con: This does not
work in Domino version
8.5.0 and 8.5.1 (SPR
JSTN825PAV)
You search an
application and
when reviewing
the resulting
documents you
observe that
When opening a document that is a
result of a search, Domino will highlight
all of the search words found within the
search. Domino will also search the
attachment for the search string to
determine whether or not the attachment
Pro: Opening
documents after a
search is faster for
users and require less
server resources.

115
when opening a
document
containing an
attachment it is
slower than
opening a
document
without an
attachment.
should be highlighted. You can prevent
this search and thus prevent the
attachment from being highlighted by
setting
FT_LIMIT_HIGHLIGHT_FILTER=1 in the
server notes.ini file.
Con: Attachment will
not be highlighted so
user may not realize the
result of their server is
contained in the
attachment.
It is desired that
temporary files
created and
used by the
indexing process
are created
outside of the
data directory.
During the indexing process the server
will need to create and manipulate
temporary files. You can specify the path
where you would like these temporary
files stored by setting
view_rebuild_dir=<complete path to
desired directory

into the server notes.ini file.
Pro: Fewer files in the
data directory will make
the server more
efficient.

Con: Administrator
must be aware of where
the files are located to
ensure the desired
storage location is
available for the server.
Need to
temporarily
disable the
chronos task.
Chronos can be disabled by setting
Debug_Disable_Chronos=1.
Pro: Occasionally may
be necessary when
troubleshooting or can
be used to compare
system performance
with and without
chronos.

Con: Chronos is
responsible for updating
view and full text
indexes set for hourly
updates. With chronos
disabled, those views
and indexes will not get
updated until the nightly
updall process runs.

3.8.3. Domain Indexes
A domain index is a full-text index over multiple Domino applications or databases. The
domidx task is responsible for building and maintaining a domain index. A domain index
can be especially helpful in legal situation where auditors may need to be able to search
your entire organization for specific data. For information on planning and creating a
domain index refer to an article that is still as relevant in Domino 8.5.x as it was in
Domino 5 entitled Domino R5: Domain Search or the Domino Administrator help topic
Planning the Domain index or the wiki article Key points to consider when creating a
domain index.

3.9. Backup a Domino Environment
116
Lotus Domino is a database system which differs from a traditional file server. On file
servers users are mainly accessing files during office hours so administrators can run a
backup during non office hours.
Lotus Domino servers instead are actively accessing their databases at all times, even
when no user is accessing the mail file or application. Because an NSF file is being
accessed at all time (in use), most operating system backup software products will skip
them. Although this is acceptable for file servers, it is not for Domino servers.

This article describes different technologies and strategies which can be used for
efficiently backing up a Lotus Domino server and avoiding headaches when you need to
restore this data. We provide information on how to define a backup plan which can
provide confidence that you can restore your Lotus Domino functions and recover critical
data as fast as possible.

To set expectations correctly, the main purpose of a backup in the context of this article is
to restore a server and its data in case of a disaster. Do not mix up backup with
archiving because they are completely different topics.
117
3.9.1. Backup Basics
As mentioned earlier, a Lotus Domino server backup can not be handled like a file server
because Domino is keeping a large number of files opened and is performing background
actions against those open files even if there is no user accessing them.

Relying on normal backup software which is not aware of how to handle open files can be
dangerous, as this software might claim a file as locked which Domino is trying to access.
This will cause the error This database is currently being used by someone else... in
Domino.

In short, the problems are:
Backup software and Domino are claiming accessing the same files. Without control,
data will be corrupted.
Downtime.

Generally there are several accepted strategies, tools and features can be used to
efficiently back up Lotus Domino.

The backup solution for Domino only covers the Domino data (including attachments)
and program files of the Domino servers to be backed up. It is beyond the scope of this
article to describe a backup and restore solution for the operating system, which is
supposed to be covered by the operating system level backup.
3.9.2. Backup Strategy
This article describes different backup strategies for Lotus Domino where clearly not
every concept will fit all environments. We assume that with a growing size of an
environment, the need for reducing downtime and increasing functional capabilities will
change.

The following table provides a suggestion as which strategy shall be applied for which
size of an environment.
Strategy Small Medium Large
Offline Backup +++ +
Clustering / Shadow
Backup
++ ++
Open File Backup + ++
Domino Application
Level Backup
+ +++ +++
Replication (~)
Offline Backup
As mentioned earlier, Lotus Domino databases can not be backed up by simple file
backup software because Domino is claiming access to the NSF files.
118
Offline backup describes a method where the Lotus Domino server is being shut down
before the backup starts and will be restarted when backup has finished. This action
ensures that the database files are not in use. Of course this method will cause downtime
for the server and therefore is not a recommended option for every environment.

In small environments where downtime is acceptable, this strategy can very well be a
considerable option because data can be backed up without using specific backup
software. Instead, simple backup software like the one used for backing up your
operating system is enough.

For automating a Lotus Domino server shutdown, administrators should use a script at
the operating system level which is scheduled to start before the backup starts. Some
backup software can also execute operating system commands as part of the backup job
itself, typically called pre- and post processing.
For a Domino server running on Microsoft Windows use the following commands:

Before Backup starts net stop Lotus Domino server
After backup has
finished
net start Lotus Domino server
Use this method when
Server downtime of several hours is acceptable for your business.
Certified backup software for Domino is not available.
Note:
Do not use offline backup method if you want a point-in-time restore. Point-in-time
restore is only possible with application level backup in combination with archived
transaction logging which is a feature of Lotus Domino.
Domino server will be down as long as backup is in progress. If the backup job
hangs, Lotus Domino will not start up without manual intervention.
Never store backup sets on the same machine that you back up. In case of a
disaster, you might not be able to recover any data!
Open File Backup
If downtime is not acceptable, a good (but expensive) method is to use a specific
enterprise software package with a feature to back up open files on Domino servers. This
type of software allows centralized backup of all the Domino servers in your network
without taking them offline.

This method does not require specific Domino certified software as the backup software
is using different technologies to back up open files.

Use this option when
Domino servers do not use transaction logging.
Downtime is not acceptable, but using a Domino certified backup software is not
possible.
You do not need to restore Domino data at a specific point in time. Otherwise, you
need an application level backup which includes transaction logs.

119
Domino Application Level Backup
This method probably describes the most common method of backing up Lotus Domino
servers, where operating system and Domino data are being backed up separately by
software which is certified to use the Lotus Domino Backup API.

An application level backup does not include any operating system files, so you need to
implement two backup systems on the same server:
One for backing up the operating system and its data.
An application level backup which is the a Lotus Domino using certified backup
software.

In a standard Notes database (NSF), the attachments are stored inside the NSF file
itself, and the database is self-contained. In order to back up a standard Notes database,
only the NSF file itself needs to be backed up. Starting with Lotus Domino 8.5, a new
feature called DAOS is introduced where the NSF files that participate in DAOS no longer
contain all data. Instead they contain only references to attachment content which is
stored separately in files of type *.NLO
As a result, backing up the NSF alone is no longer enough. The NLO data needs to be
backed up as well. For more details, refer to IBM Technote 1358548 - http://www-
Note:
Do not forget that important data is stored within the Lotus Domino program directory.
For example, the NOTES.INI and the Server.ID should be part of a regular back up!
Transaction Log Backup
Transaction log backup is actually not a strategy itself. It is an extension to the Domino
application level backup, because in order to take a benefit of transaction log backups,
administrators must perform the restore with a software which is certified to work with
Lotus Domino.

Transaction Logging is a process which captures database changes and writes them
sequentially to a so transaction log, which should be on a separate disk drive (not just a
partition!). Domino builds transaction log files on this drive and does not delete them until
the 3rd-party backup software backs them up using API. Transaction logging improves
performance and ensures data integrity of databases by committing to the transaction log
first before writing to the Notes database.

Transaction logs in Domino can be configured in two modes:, circular and archived
mode.
Archive mode - The transaction logs (*.txn) must be backed up with certified Domino
backup software, for example IBM Tivoli Storage Manager for Mail, Data Protection
for Lotus Domino. If no backup is taken, then the log files will grow further and further
until the transaction log disk drive is full and your server will crash. Ensure that you
have enough disk space available to support a time frame of at least 24 hours without
any backup.
120
Circular mode - Use a configurable fixed amount of disk space to improve the server
performance. It does not offer to restore a database at any point-in-time. In this
mode, you can only restore the database as it was when you took the backup.

More information about transaction logging can be found in the following documents:
IBM Technote 7003543 Transaction Logging on Domino Servers
IBM Infocenter Article Setting up a server for Transaction Logging
IBM Technote 7009309 Best Practices Transaction Logging

Replication
Probably the least efficient method for backing up data is replication where
Administrators configure Lotus Domino to replicate important databases across different
servers to ensure that a live backup of the information is available somewhere else.

This method requires much planning because:
Deleted documents can and will replicate. You cannot restore a document which was
just deleted by accident. You can configure the database to not replicate deletions.
This adds a significant management overhead. In addition, it might not acceptable for
all databases.
Design elements can and will replicate. This causes design and data corruption
which in the end might make the backup useless.
Critical files which are local to the server itself are not replicated. For example,
NOTES.INI and Server ID files are not replicated thus backed up. If a servers data
drive fails and you do not have a backup of these files, then the system is in trouble.

Never rely only on replication as your method of database backup. A damaged or
accidentally changed database may replicate, and then your only recourse is to recover
the database from a server backup tape. This method must be combined with another
backup concept in order to provide any value.
Cluster, Shadow Copy, and Shadow Backup
This backup strategy is a mixture between Replication and Offline backups.
Assumption: Both servers are hosting the same data and replicate on a regular basis, or
even are part of the same Domino cluster (as shown in the example below).

During office hours, both servers are active and users can work on both instances.

During the non office hours, Server B is taken offline and is backed up using the offline
backup method described earlier. While backup is in progress, ServerB is down. During
this time, users can still access Server A because it is part of the same cluster. Users can
continue to do their work and changes will replicate back to Server B once Server B is
started up again.
121

The challenge in this strategy is to keep both servers synchronized because not every
database created on Server A automatically has a replica on Server B. Furthermore, all
data must successfully replicate from one server to the other, which for example, is not
the case if applications use reader name fields where servers are not a member of.

Additional and more advanced options are offered by Enterprise Storage Systems where
similar strategies are used. First, the online data is being mirrored to a dedicated storage
pool. To perform a backup, the flash copy process is interrupted as long as the backup is
in progress. Describing this technology and the different vendor options is beyond the
scope of this article. Consult your storage specialist for more details.

Note:
Lotus Domino currently does not support Microsoft Volume Shadow copying. For
further details, refer to:http://www-
To understand how Domino clustering works, refer to this old (but valid) IBM
Redbooks publication:
3.9.3. Backup Software
The backup software you use for your Domino servers depends on the backup
infrastructure back-end in your environment. Although there are a variety of certified
backup products available for Lotus Domino on the market, we focus on the following
IBM products within this article:
IBM Tivoli Storage Manager
http://www-01.ibm.com/software/tivoli/products/storage-mgr/
IBM Tivoli Storage Manager for Mail, Data Protection for Lotus Domino (further
described as TDP) http://www-01.ibm.com/software/tivoli/products/storage-mgr-
mail/
Which is certified for backing up Lotus Domino data incl. transaction logs
To perform an application level backup of Lotus Domino, the following components are
required:
IBM Tivoli Storage Manager Server (TSM Server)
IBM Tivoli Storage Manager Agent (TSM Client) for static data backup and restore
including DAOS
TSM for Mail Data Protection for Lotus Domino (TDP) 5.5.2.1. See description
below.

122
Tivoli Storage Manager for Mail - Data Protection for Lotus Domino
(TDP)
TDP provides the function to do online backups (this means without the need to stop the
Domino server) of Domino databases and transaction log files. TDP only acts on
databases (*.ns*), templates (*.ntf), and transaction log files (*.txn). TDP also handles the
restore of data. TDP communicates with the TSM server via the TSM application program
interface - the TSM client. For the communication with the Domino server, TDP uses
Lotus Domino API. There are several ways to operate TDP: command line interface, GUI,
and integration within Tivoli Manager for Domino.
For more information about system requirements of TDP, refer to IBM Technote
1297052.
Note:
TDP offers no functions for disaster recovery of a Domino server because only the data is
backed up and not the program files. To support disaster recovery, TDP and the TSM
client must be used in combination.

TDP and the Interdependences with Domino Server Tasks
On a Domino server with transaction logging enabled, Domino assigns a Database
Instance Identifier (DBIID) to each Domino database. When Domino records a
transaction in the transaction log file, it includes the DBIID. During restore, Domino uses
the DBIID to match transactions to databases. There is no relation between the Replica
ID and the DBIID.

There are two database maintenance activities:
COMPACT (all switches beside "-b")
FIXUP, which causes the Domino server to assign a new DBIID to a database
From that point forward, all new transactions recorded in the transaction log file use the
new DBIID. However, any old transactions still have the old DBIID and no longer match
the database's new DBIID. As a result, Domino cannot restore the old transactions to the
database. In this scenario, TDP must be used to make a new backup of this database.
That is done by the incremental type of TDP backup.
3.9.4. What to back up
The table below shows which data must be included in the Domino backup. The
operating system files are not included in this list as they vary between different
operating systems.
# Data Description Backup
Utility
1 Program
code
Backup of the Domino program code
\*.*
File
backup
e.g. TSM
client
2 Static data Backup of the Domino data directory with
all subdirectories and directory links.
File
backup
123
\*.*
The following files and directories must
be excluded:
o .ntf (templates)
o .ns* (databases)
o .box (mailboxes, SMTP
mailboxes)
o .txn (transaction log files)
o .txn.dad (transaction log files)
o .ft\ (full-text index directories)
o .tmp (temporary files within the
Domino Data directory)
e.g. TSM
client
3 Incremental
backup
Backup of new Domino databases,
databases excluded from transaction
logging, and databases changed through
a FIXUP or COMPACT program
document.
Certified
Domino
Backup
utility, e.g.
TDP using
incrementa
l option
4 Domino
database
backup
Backup of system databases, templates,
mail files and application databases
Certified
Domino
Backup
utility, e.g.
TDP using
selective
(full
backup)
5 Transaction
log backup
Backup of transaction log files
Daily multiple (every 2 hours) for
Transaction log files from type archive.
No backup for transaction logs from type
circular
Certified
Domino
Backup
utility, e.g.
TDP using
archive log
option
6 DAOS data Backup of DAOS data provided on DAOS
drive.
According to weekly and monthly full
backup
File
backup
e.g. TSM
client

3.9.5. Backup procedures
There are different kinds of backup that are handled by Domino certified backup
software:
Full backup
Incremental backup
Selective backup
124
Archive log
Apart from the application level backup, one lower level type of backup must be
mentioned:
DAOS Backup
We describe these backups in the following sections.
Full Backup
A full backup is a backup that includes all files that can be backed up using Domino
certified backup software. Unlike a file based full backup, a Domino certified backup
software can backup Domino files while Domino is online.

Incremental Backup
Online backup of complete databases via TDP where at least one of the following
conditions have to be fulfilled:
The database is not excluded from the backup by rules (include-exclude-lists).
The database is enabled to use transaction logging and the DBIID is changed. All
database transactions use the DBIID as key which is unique for a database on a
server. The DBIID of the transactions in the log must match the DBIID of the
database so that the recovery of the transaction logs is possible. The run of the fixup
or the compact task changes the DBIID. Only if compact is started with the "-b" option
(in-place compaction), the DBIID stays unchanged.
The database is not included in the transaction logging and is modified since the last
backup. Changes in the database itself or in the access control list are used to
determine the backup of that database.
The database is new or is just added to the backup list.
Important: The incremental backup function of TDP should not be mixed up with the
rules valid for the TSM client's incremental backup, like backing up file if the modification
date and time of the file is changed. TDP backs up all databases which fit one of the
conditions mentioned above. Both, the incremental backup and the selective backup of
TDP are full backups. This means that both backup operations store complete
databases.
Selective Backup
The selective backup selects that data out of the data pool of Domino that will be backed
up. Use this feature to backup all files that can be backed up using TDP. Leave out the
cluster member mail files. This helps saving space.

Archive Log
The archive log with TDP stores filled transaction log files on the TSM server so that
space allocated to these files can be reused by the Domino server. The archive log
command is available if archival transaction logging is enabled on the Domino server.
Filled transaction log files must be archived frequently enough to ensure the transaction
log storage space is never allocated completely and stops the Domino server.
Transaction log files stored on the TSM server are automatically restored as needed for a
125
database recovery. Archived transaction log files are retained on the TSM server as long
as a database backup exists that needs these log files for a complete recovery.

All the backup procedures mentioned above can be executed automatically using
command line scripts.

DAOS Backup
DAOS backup is done according to the schedules of TDP full backups (weekly and
monthly basis). The data should be written to the same management classes that are
being used for Domino backup and therefore has the same retention periods. DAOS
backup should be done using the TSM Client so it backs up only the files that have been
changed (therefore it is incremental).
For more information on DAOS backup backup and recovery, refer to this Tivoli Field
Guide:

3.9.6. Backup Scripts
This section represents an example backup schedule for an environment where Domino
Data needs to be restorable for 12 months.

The following table contains steps to back up various files types, with their data retention
settings, the script names, and the generic schedule settings for the scripts on the
Domino servers:

# Data Pattern Retention Script / Schedule
1
Program
code
Domino program
code
*.*
3 days
Included in the daily
incremental TSM
backup of the OS
2 Static data
Static Domino
data files,
Domino data
directory with
subdirectories
and directory
links.
\*.*
Exclude:
o .ntf
o .ns*
o .box
o .txn
o .txn.dad
3 days
Included in the daily
incremental TSM
backup of the OS
126
o .ft\
3
Incremental
backup
Domino data
files
30 days
incremental.cmd
/daily
4
Domino
database
backup
Domino data
files
6 weeks
dombackupWK.cmd
/
Weekly (with the
exception of the last
weekend of each
month)
5
Domino
database
backup
Domino data
files
12
months
dombackupMT.cmd
/
weekly (the last
week-end of each
month)
6
Transaction
log backup
Transaction log
files
30 days
archivelog.cmd /
daily or multiple
times per day (every
2 hours)
7
DAOS
backup
weekly
DAOS Files 6 weeks
dombackupWK.cmd
/
8
DAOS
backup
weekly
DAOS Files
12
months
dombackupMT.cmd
/
weekly (the last
week-end of each
month)
9
No backup of
files. This
command
inactivates *.txn
on the TSM
server.

inactivatelogs.cmd /
weekly included in
dombackupWK.cmd
/
weekly
Note, the steps 6 through 9 are valid only for Domino servers with archival transaction logging
enabled.
3.9.7. Recommendations
Here are some tips which can help in any environment:
Publish your backup standards so that end users know when and how often a backup
is taken.
Perform backup restoration tests to ensure valid recovery data.
Define what data you store for how long, and carefully think about what happened
afterwards. The oldest backup is what you are able to restore. Evaluate if it makes
127
sense to take a yearly backup (acting as an archive version) which is stored in a safe
place.
Ensure your backup environment is properly monitored and follow up any errors as
soon as possible. Document when an error caused a restore to be not available.
Work to institute charge-back to end-users for backup resources consumed. This
way, you can have your users understand the cost of a good backup infrastructure.
Understand the broader backup environment which includes the operating system
below of Domino as well as the backup back-end, for example, tape robots, and their
schedules etc.
Collaborate with other teams. share your knowledge with other people who are not
familiar with the Domino platform and how backup is done within.
3.9.8. Summary
Performing Lotus Domino backup need to fulfill requirements similar to other application
systems like an SQL database. The most efficient backup concept depends on the size of
your environment and the provided backup infrastructure (if any).
3.9.9. Reference Reading
Information provided within this article is based on the following documents:
Lotus Knowledge base Technote #7009311"Notes/Domino Best Practices: Backup &
Recovery"
http://www.ibm.com/support/docview.wss?uid=swg27009311
Lotus knowledge base Technote 1358548 DAOS Backup and Restore
IBM InfoCenter
http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/com.ibm.help.domino.ad
min85.doc/H_ABOUT_BACKING_UP_THE_NOTES_SERVER.html
Lotus Knowledge base Technote #1196479 Is Microsoft Volume Shadow Copy
supported for Domino backups?
IBM Knowledge Base Technote #1095238 Some considerations on the use of
TSM/TDP as a backup solution for Notes/Domino
IBM Redpaper Choosing the right backup strategy for Domino 6 for iSeries
http://www.redbooks.ibm.com/abstracts/TIPS0377.html?Open
IBM InfoCenter IBM Tivoli Storage Manager for Mail Data Protection for Lotus
Domino
http://publib.boulder.ibm.com/infocenter/tivihelp/v1r1/topic/com.ibm.itsmfm.doc/dpdo
mw.htm
3.10. Restore
Within a Lotus Domino environment, there are different restore scenarios:
A full system restore - Which is required to recover from a disaster like a complete
hardware or site failure.
A restore of one or more database - Replacing the existing one or more databases.
Restore of one or more documents within a database.
128

In general, the data restore technology depends on the scenarios. Also, the server type
and the backups available are of interest.
The following table lists different type of data and the tools involved in the restore
process:
Data Type Utility
Program files File backup software, e.g.TSM client
Static Domino data files File backup software, e.g.TSM client
Domino data files Certified Domino Backup utility, e.g. TDP
In addition to the table above, Domino servers where transaction logging is enabled
requires:
Data Type Utility
Transaction log files Your certified Domino Backup utility, e.g. TDP
DAOS data Your file backup software, e.g.TSM client
DAOS enabled transaction logs require a different tool to backup the DAOS storage
directory. DAOS stores attachments separately from the database files, and are not
backed up with the traditional backup and restore products.

Transaction log files use a product external to Domino for backup and restores. Without
transaction logging, DAOS and NSF may both use flat file recovery.

Three parts of a restore process can be distinguished:
Disaster recovery
Static file recovery
Domino data file recovery

3.10.1. Disaster Recovery
This process is to restore a whole Domino server by recover all files and directories. A
prerequisite for this process is to have a fully recovered operating system with all registry
and configuration settings.

One disaster recovery scenario is to perform a bare-minimum-restore, which means to
recover all software to a blank piece of hardware. Here the main challenge is to restore
the operating system so that it is fully functional as before. How to do this depends on
your servers operating system and your configuration. Consult the operating system
vendor as well as your backup software vendor for more detailed instructions.
If you can not bring the original server back online, for example, because of an hardware
failure or similar problems, consider restoring Lotus Domino to a different hardware.
Domino itself is not forced to use the same operating system or the same hardware.
Lotus Domino can be moved if you take some items into consideration. IBM Technote
1087009 http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg21087009
describes how to move a Lotus Domino server from one hardware to anther. The same
129
concept can be used in case of a disaster recovery to rebuild the server on a different
hardware than before.
As soon as operating system and installed operating system applications are recovered,
it is time to restore Lotus Domino. For this to be done, tools and processes used for
disaster recovery are described in the next two sections.
3.10.2. Static File Recovery
Static files are Domino program files and files in the Domino data without Domino
databases and templates. In our example, the TSM client will be used to restore these
files.

It is possible to restore the most recent backup or a file from a specific date, given that it
is still retained within the backup. This restore process is not different than restoring a file
from a file server.

Because Domino is constantly using some of the files within the program directory, when
restoring Static Domino files, consider shutting down Domino before restoring the static
files.
3.10.3. Domino Data File Recovery
Individual restores might be requested from users, for example, to restore a particular
database (.nsf file) for a specific point in time from the backup environment. Users
typically expect to get access to the data in the same way as before.

To accomplish different data restore, the following strategies are available:
Stand-alone - Provides the restore in the form of a stand-alone application. It is
accessible for users. The user needs to find and restore individual Notes documents.
Merge - Restore data back into production by merging current data with the restored
data.
Replace - Replace the current application completely with the .nsf file obtained from
the backup.

Each of these options is described in the following sections.
Stand-alone Restore
For a stand-alone restore, Domino Administrators provide the restored data so that the
user can get access without impacting the data in production.

Although it sounds simple, there are small challenges which administrators need to be
aware of for the restored databases:
Restored databases are not any different from production databases when it comes
to their functionality.
Restored databases can and will run scheduled background agents (if not explicitly
disabled).
Have the same ACL and will therefore immediately allow users to access this file (if
not configured otherwise).
130
Restored database can replicate (if not configured otherwise).
For restoring Domino data files, TDP/TSM is used in this example. On Lotus Domino
servers without transaction logging enabled in archive mode, TDP recovers one or more
designated files from the TSM server.

In general, replication will be disabled for this restored database by the restore process
(TDP/TSM) automatically, so restoring back to the server where files have been backed
up is the preferred method, however there are cases where this is not possible e.g.
because an old server has been switched off.

The Domino data file recovery is a three-stage process:
1. Restore one or more data files. It is possible to restore the files under a different
name, in a different directory or to a different server. (See figure 1 below). Domino
data file restore can cover the most recent versions or a specific date for the files to
recover.
2. Activate the Domino data files. This function brings restored databases online for use
by the Domino server. It is optional to apply transactions from the transaction log to
update the database. Transactions can be applied up to a specific point in time or up
through the most recent changes recorded in the transaction log. If archival logging is
in effect, TDP for Lotus Domino automatically restores archived transaction log files
as needed.
3. After transaction log restore is completed and Domino database restore is therefore
complete, TDP provides a list of DAOS Files that must be restored to complete the
restore. Restore of the DAOS files is done using the TSM Client.
TDP restores Domino data files at the database level. To restore single documents, the
entire database must be restored. Afterwards the documents can be copied to the
original database using the Notes client.
Perform Single Database Restore
To prevent the most common mistakes, figure 1 illustrates a flowchart that provides
assistance for defining the most efficient restore method and defining your restore target.

Note: The flowchart should be used as a reference only. It should not be mapped onto
your environment as-is because every environment is unique and there are many things
need to take into account that we might not mentioned here.

131

132

Number Comments and considerations
1 Is the database you are trying to restore encrypted?
For details see the database properties and encryption settings.
2 On the server where the backup was taken, is (or was) DAOS enabled on
this server?
Check the server document DAOS tab for details
3 On the server where the backup was taken, is (or was) DAOS encryption
enabled?
Note: DAOS Encryption is enabled by default unless you have explicitly
configured your server to not encrypt DAOS objects (*.NLO files). This can
be beneficial if you want to be able to restore back to server other than the
restore was taken from.
4 In this situation, you must restore the database back to the server where
the backup was taken. The server ID must be the same because the
database itself or its DAOS objects are encrypted with this server ID.
Restoring to a different server is not possible.
5 Although it is more efficient to restore a database back to its original
server, there are environments where corporate policies request the usage
of a dedicated restore server.
In this scenario, you can if you want - restore this database to a different
server.
6 Restore the Domino NSF file(s) using your backup and restore software.
Make sure all of the following items are checked before you continue:
Change the replica ID. Some backup software can do that as part
of the restore process. If not, you have to do this manually!
Disable replication. Some backup software can do that as part of
the restore process. If not, refer to IBM Technote 1094568
Disable scheduled agents, by enabling this check box in the
database properties of the restored file.

133
(Optional) change the ACL of the restored file so that only the
requestor can get access.
7 Restore DAOS Objects. For more detailed actions, refer to this
DominoWiki article http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daos-
backup-and-restore#Restoring+DAOS+objects
8 Open the restored database (or request users to do that) and restore the
documents requested.
9 If the restored database(s) are no longer required, delete them from the
server.

Merge Restore with Production
Once a database has been restored using the method described above, you might want
to merge data restored with the one currently available in production.
Clearly, merging a restore with production is NOT a recommended method because
documents which have been deleted in the production replica will leave a deletion stub.
So replicating a restored version of the same database will NOT restore the deleted
document.
Although those deletion stubs can be removed, there might be local replicas of the same
database for example, on the users desktop which also contain those deletion stubs.
Removing deletion stubs as described in IBM Technote 1095683 will not remove them in
other replicas. Result: Deletions will be performed again once you clear the replication
history.
Replace Production Database with Restore
If you need to replace a production mail file or application database with a restored
version of the same file, you can of course delete the production database and store the
.nsf file which was restored as a single database restore (see above) to the same folder
and file name.

Note this will not work if:
You are attempting to restore a Domino system database; these files are in use by
Domino and often require the ReplicaID to be consistent within a Domain. For
technical details, refer to IBM Technote 1099635.
As part of the restore process, you have changed the ReplicaID, but some (badly
developed) custom applications require a specific ReplicaID to be used. Before
changing the replicaID as you want it to have, think twice about the consequences!
Any user who got a replica of this file will replicate old content back to the server,
including modifications you might not want to have.
When replacing a production database with a restore, evaluate using a different replica
for the restored version. This prevents from old or modified content to replicate back into
the restore.
134
Optimizing the Restore process
A restore process which is optimized for providing server availability and user satisfaction
would run as an half automated process which can look like this:
Restore the ,nsf file to a place outside of the Domino Data directory.
This will ensure the file would not replicate with other servers.
A server side scheduled agent would open the database from this path outside the
Domino Data directory and set the DB property Disable background agents in this
file."
The source code for such an agent can be found in IBM Technote 1380020. As an
alternative, disable all scheduled agents as described in the end of IBM Technote
1201461
To put back the database online and change the replica ID at the same time, use the
Domino server command Cluster Copy , for example
> CL COPY C:\Restore\Filename.nsf RESTORE\filename.nsf
This command creates a non-replicating copy of the database
C:\Restore\Filename.nsf in the new location Restore\filename.nsf below of the
Domino Data directory.
For more details about this command and its usage, refer to this DominoWiki article
Note: for this command to be available, make sure to set the NOTES.INI variable
CLUSTER_ADMIN_ON is set to 1
(If required) change the ACL as needed.
Recommendations
In general, the following recommendations should be kept in mind when performing
restores:
If you want to use a dedicated server for your restores, consider to disable DAOS
Encryption as described in this DominoWiki article
http://www-
10.lotus.com/ldd/dominowiki.nsf/dx/DAOS_Deployment_Guide#Deciding+on+encrypt
ion+for+NLO+files
Define AND test your emergency restore procedures on a regular basis.
When restoring custom developed Domino applications, consult the application
developer for specific details on how to restore individual documents without
impacting the applications logic.
Consider using specific file names which clearly indicate the point in time of a restore.
For example, . Restore\2010-11-02\filename2009-12-01.nsf indicating the file
filename.nsf was restored on Nov. 2
nd
2010 as it was on 1
st
of December 2009. Feel
free to use your preferred method of naming files and folders, but make sure to use a
consistent approach as this naming can be used for a cleanup script later on.
Delete restored files when they are no longer required.
This can be automated with a small script if you define that any restore will be
removed (e.g.) 7 days after it was provided.
Configure your Domino servers to enable Cluster Administration command line
options.
For more details, refer to this DominoWiki article.

135
3.11. Procedure to Restore Deleted Documents on IBM i
This article assists you when trying to recover documents that were accidentally deleted
within a Domino application or database.

It is assumed that you will be restoring a copy of the database prior to the document
deletions. The restored database will be placed on the server and the user will then
access the restored copy of the application to copy and paste the documents back into
the existing database.

There are a number of considerations that should be made when performing this type of
restore:
DBIID: You must ensure that you will not have 2 databases on the server with the
same DBIID because multiple databases with the same DBIID could damage your
transaction logs.
Replication: You need to prevent any replication attempts to the restored database.
To accomplish this you must prevent access to the database or ensure the database
replica ID is changed during the restore process.
Scheduled agents: If the application being restored has agents that normally would
run on this server, you want to ensure these agents are disabled as part of the
restore process to prevent the agent from running on this temporary file.
Since there are different restore requirements and/or limitations depending on the type of
save you performed, figure 1 below and the subsequent explanations walk you through
the restore process with these factors already considered for you. The complete steps
and commands can be found in the appropriate sections below the diagram.
136

3.11.1. Operating Type Save Restore Procedure
1. Gather the tape(s) from last save of the database/ application.
2. Restore database to a folder outside of the data directory:
If you do not have a restore directory already created, create one now. For example:
CRTDIR '/Restore'
Restore the database to the new folder. For example:
RST DEV('/qsys.lib/tap01.devd') OBJ(('/notes/data/mail/mailfile.nsf' *INCLUDE
'/restore/mailfile.nsf'))
Set permissions so the QNOTES user profile can access the directory and object. For
example:
CHGOWN OBJ('/Restore') NEWOWN(QNOTES) SUBTREE(*ALL)
3. (Optional) Use an agent to disable scheduled agents from running on the database. A
server side scheduled agent would open the database from this path outside the Domino
Data directory and set the DB property Disable background agents in this file. The
137
source code for such an agent can be found in IBM Technote 1380020 or as an
alternative simply disable all scheduled agents as described in the end of IBM Technote
1201461
4. Create a restore folder in the server data directory. In the Domino Administrator go to the
Files tab. In the right hand panel select Folder New.... Give the folder the desired name
such as Restore.
5. Use the CL Copy command to copy the database to the Restore folder within the data
directory. Using the CL copy command will ensure that the database will have a new
DBIID and replica id when placed on the server. The Domino console commands to issue
are as follows:

set config CLUSTER_ADMIN_ON=1 this will enable the use of the CL Copy command.
CL COPY /restore/mailfile.nsf restore/mailfile.nsf Note that the initial entry is the
source database with the complete path and the second entry is the destination with a
path relative to the data directory.
6. Verify integrity of restored database and DAOS links by running fixup. For example:
load fixup -j restore/mailfile.nsf

Note: The fixup can be skipped if running 8.5.1 FP2, 8.5.2 or higher and restoring to the
same server (refer to SPR DROO7YXTC3). The fixup process is critical when restoring
to a different server as the fixup process is needed to update the .nlo file hints stored
within the documents of the database.
7. Copy and paste the needed documents into the existing database. If you get an error
regarding a missing .nlo file you can get the file name from the error message posted in
ddm.nsf or generate a list of the .nlo files needed to be restored using the "listnlo"
command. For example:
tell daosmgr listnlo -o missing_nlo_files.txt MISSING restore/mailfile.nsf
8. Review output from the "listnlo" command above and restore any files listed. Note that
the same file may be listed multiple times in the output, one for each reference to the
missing attachment. The syntax for the restore is the same as step #2. For example:
RST DEV('/qsys.lib/tap01.devd')
OBJ(('/notes/DAOS/0001/AAC48E7FC07B6CC71ADD186896DC4F48509F165801A8
702F.nlo'))
9. Cleanup by remove the restored database. Remove the copy within the data directory
using the Delete Database right click menu option in the Domino Administrator Files
tab. To delete the file outside of the data directory use the following command:
RMVLNK OBJLNK('/restore/mailfile.nsf')
The restore directories may be retained for reuse or removed at your discretion.

3.11.2. BRMS Full Save Restore Procedure
1. Gather the tape(s) from last save of the Domino server.
2. Create a restore folder in the server data directory. In the Domino Administrator go to
the Files tab. In the right hand panel select Folder New.... Give the folder the
desired name such as Restore.
3. Set a directory ACL to prevent all users and servers from accessing the directory.
This will prevent replication and thus prevent propagating deletions to this newly
restored database. To enable this feature you must set enable_acl_files=1 in your
NOTES.INI. To do this, enter the Domino console command set config
enable_acl_files=1. You can then return to the Domino Administrator Files tab, right
click on the folder and choose Manage Directory ACL. From there choose an
administrator to be able to access the directory, but no other users or servers as
138
shown in figure 2.

4. Use RSTBRM, WRKLNKBRM, WRKMEDIBRM, or System i Navigator to restore the
database to your restore folder. Here is an example of RSTBRM:
RSTBRM DEV(*MEDCLS) OBJ(('/server1/data/mail/mailfile.nsf' *INCLUDE
'/server1/data/restore/mailfile.nsf'))
5. Change the Replica ID of the restored database using CL Copy, an agent or 3rd
party utility such as Antrid. For information on changing the replica id refer to
technote 1094568. Here is an example of using CL Copy:
set config CLUSTER_ADMIN_ON=1 this will enable the use of the CL Copy
command.
CL COPY restore/mailfile.nsf restore/mailfile2.nsf Note that the initial entry is the
source database with the complete path and the second entry is the destination with
a path relative to the data directory.
Remove the copy that the database with the original replica id using the Delete
Database right click menu option in the Domino Administrator Files tab.
6. (Optional) Disable any scheduled agents in the restored database manually or via an
agent. A server side scheduled agent would open the database from and set the DB
property Disable background agents in this file. The source code for such an agent
can be found in IBM Technote 1380020 or as an alternative simply disable all
scheduled agents as described in the end of IBM Technote 1201461.
7. Remove the directory ACL. To do this return to the Domino Administrator Files tab,
right click on the folder and choose Manage Directory ACL. From there remove all
entries from the Who should be able to access this directory list.
load fixup -j restore/mailfile2.nsf

Note: The fixup can be skipped if running 8.5.1 FP2, 8.5.2 or higher and restoring to
the same server (refer to SPR DROO7YXTC3). The fixup process is critical when
restoring to a different server as the fixup process is needed to update the .nlo file
hints stored within the documents of the database.
139
regarding a missing .nlo file you can get the file name from the error message posted
in ddm.nsf or generate a list of the .nlo files needed to be restored using the "listnlo"
tell daosmgr listnlo -o missing_nlo_files.txt MISSING restore/mailfile2.nsf
10. Review output from the "listnlo" command above and restore any files listed. Note
that the same file may be listed multiple times in the output, one for each reference to
the missing attachment. The syntax for the restore is the same as step #2. For
example:
OBJ(('/notes/DAOS/0001/AAC48E7FC07B6CC71ADD186896DC4F48509F165801
A8702F.nlo'))
11. Cleanup by removing the restored database. Remove the copy within the data
directory using the Delete Database right click menu option in the Domino
Administrator Files tab.

3.11.3. BRMS Incremental Save Restore Procedure
1. Gather tape from the server saves starting with last full save prior to the desired
restore date along with each incremental from the full save to the desired restore date
of the Domino server.
2. If you need to recover to a point in time between now and last save, run an
incremental (transaction logs only) save now.
3. Create a restore folder in the server data directory. In the Domino Administrator go to
the Files tab. In the right hand panel select Folder New.... Give the folder the
desired name such as Restore.
4. Create the required data area to force the replica ID to be modified during the restore
with the following command:
CRTDTAARA DTAARA(QUSRNOTES/QNNIDIFFID) TYPE(*CHAR) LEN(1)
The data area needs to be owned by QNOTES. Set the owner with the following
command:
CHGOBJOWN QUSRNOTES/QNNIDIFFID *DTAARA NEWOWN(QNOTES)
5. Use RSTBRM, WRKLNKBRM, WRKMEDIBRM, or System i Navigator to restore the
database to your restore folder. Here is an example of RSTBRM:
STBRM DEV(*MEDCLS) OBJ(('/server1/data/mail/mailfile.nsf' *INCLUDE
'/server1/data/restore/mailfile.nsf'))
6. (Optional) Disable any scheduled agents in the restored database manually or via an
agent. A server side scheduled agent would open the database from this path and set
the DB property Disable background agents in this file. The source code for such an
agent can be found in IBM Technote 1380020 or as an alternative simply disable all
scheduled agents as described in the end of IBM Technote 1201461.
load fixup -j restore/mailfile.nsf

Note: The fixup can be skipped if running 8.5.1 FP2, 8.5.2 or higher and restoring to
the same server (refer to SPR DROO7YXTC3). The fixup process is critical when
restoring to a different server as the fixup process is needed to update the .nlo file
hints stored within the documents of the database.
regarding a missing .nlo file you can get the file name from the error message posted
in ddm.nsf or generate a list of the .nlo files needed to be restored using the "listnlo"
140
tell daosmgr listnlo -o missing_nlo_files.txt MISSING restore/mailfile.nsf
9. Review output from the "listnlo" command above and restore any files listed. Note that
the same file may be listed multiple times in the output, one for each reference to the
missing attachment. The syntax for the restore is the same as step #2. For example:
OBJ(('/notes/DAOS/0001/AAC48E7FC07B6CC71ADD186896DC4F48509F165801A
8702F.nlo'))
10. Cleanup by remove the restored database. Remove the copy within the data directory
using the Delete Database right click menu option in the Domino Administrator Files
tab.
3.11.4. Additional Resources
How to move a Domino server on IBM i from one system to another
Summary of Backup and Recovery recommendations when using DAOS on IBM i
Lotus Server Backups documentation for Backup Recovery and Media Services (BRMS for
IBM i)
3.12. The Domino HTTP Server
Domino is more than a mail server. It can also be a powerful web server. With no
additional work or configuration, any document on a Domino server may be viewed by
the Lotus Notes client or a browser as the server automatically converts the document to
HTML. As a Domino administrator, you want to know how to effectively manage your
Domino server on the web. This article provides references to the existing information on
Domino web server configuration.
3.12.1. General Server Configuration
Before you start working with the Domino server or if you take over the administration
responsibilities for an existing Domino domain, you should be aware of some
fundamental concepts.
Domino HTTP Server Security
Building Web applications in Domino 6: A tutorial on Web site addressing
Building Web applications in Domino 6: Accessing and protecting the file system
Building Web applications in Domino 6: Web site rules

3.12.2. iNotes
Lotus iNotes is the method of accessing Domino mail via a web browser and is extremely
powerful. There are many resources available for you as the Domino Administrator to
assist with configuring and managing iNotes users.
Getting started with Lotus iNotes
Setting Up a Redirection Application for Lotus iNotes Users
Securing Lotus iNotes
141
How to rename an iNotes (DWA) user with a Notes ID
Notes.ini settings for iNotes
Customizing iNotes
Knowledge Collection: Directory Assistance and Lotus iNotes/Domino Web Access(DWA)
Lotus iNotes Administration
Adding widgets in iNotes
Type-Ahead is on my default starting with iNotes at 8.5.1
iNotes and Sametime
Setting up Lotus iNotes with Sametime
iNotes_WA_MacIM=1 required for Sametime integration from iNotes on a Mac (OSX operating
systems)
3.12.3. Troubleshooting and Tuning
Tuning Tips for the Domino HTTP Server
Troubleshooting Lotus Domino hangs and crashes
How to enable HTTP Request logging in Domino
Collecting data for HTTP crash on a Lotus Domino server
Collecting data for HTTP memory crash on Lotus Domino server

3.12.4. Additional References
As you become more familiar with using Domino as a web server, you may want to start
using it for more or new applications. Here are some references to get you started:
XPages
Building Domino Web Applications
3.12.5. Some Tips
Internet site: Use IP address and host name. This is not supported with Sametime
until version 8.5.1.
SSO: Do not change name of LTPAToken.

142
3.13. Domino HTTP Server Security
HideTable of Contents

If you have secured your Domino data for use with the Lotus Notes client, then your data
is also secured when accessed from a browser. However; there are additional
considerations after you enabled the Domino web server (http task). This article assists
new Domino administrators with basic security recommendations and concepts when
granting internet access to your Domino server. Items included are how to enforce server
access settings and controlling anonymous access as well as links to great resources for
Internet password lockout, SSL and more.
3.13.1. Server Access

The HTTP server honors the database access control list as well as document access
such as readers and authors fields. When authenticating via HTTP, a Notes certificate is
not required. In order to be able to access Domino information via a browser, the user
must have a valid person document with an internet password or token if using single
sign on (SSO). The user must also be allowed access to the server. By default, anyone
listed in the Domino directory can access the Domino server via HTTP, but this can be
controlled via the server access settings. Specifically in the server document, security
tab, you can define who can access the Domino server as shown in figure 1.

143

144
By default, these settings are not honored by HTTP. To force HTTP to honor those
settings, set Enforce server access settings to Yes in the Ports Internet Ports Web
tab of the server document as shown in figure 2. You can also choose whether or not to
allow Anonymous access to the server.

3.13.2. User Security and Authentication
To protect the integrity of your users and their passwords, Domino provides you with an
"Internet Password Lockout" function. This allows the system to "lock out" a user after a
certain number of failed log in attempts. For more information on Internet password
lockout refer to the article Securing an IBM Lotus Domino Web server: Using the new
Internet lockout feature.
145

The Domino server caches user names and passwords for 2 days. For that reason, you
may observe that when the internet password is changed there is a period of time when
both the old and new password will be accepted. If this is unacceptable in your
organization, you can control the cache by using the NOTES.INI setting
HTTP_Pwd_Change_Cache_Hours=<# of hours>. You should be aware that restarting
the HTTP task will rebuild the cache and thus cause the server to no longer accept the
old password no matter how many hours are specified by
HTTP_Pwd_Change_Cache_Hours. If you have multiple web servers, you must also
consider your replication topology as it may take some time for the new password to
replicate throughout your environment.

Many times, you may want users that are not defined directly in your directory
(names.nsf) to be able to access data on the web. Alternately, you may have a user
directory already configured that is used throughout your enterprise. Domino provides
this functionality through directory assistance. For information on directory assistance
refer to How to set up Directory Assistance in Domino or 3.4 Multiple Directories.

Domino has multiple authentication types. You can choose to enable session
authentication to minimize the number of log-in prompts presented to the user at both a
single server and multi-server level. Here are some resources related to authentication
and single sign on (SSO):
Name-and-password authentication for Web clients
Preventing multiple password prompts in Lotus iNotes
Webserver Authentication Troubleshooting
Session-based authentication (single sign-on)
How the Domino HTTP session authentication configuration affects which login prompt is sent
to Web browsers
Deploying Windows single sign-on for Web clients (SPNEGO) in an existing Domino
environment
DWA with Sametime integration
Hints and Tips for Troubleshooting Single Sign-on and Authentication Issues with Domino and
WebSphere
Troubleshooting WebSphere Portal, Domino Extended Products, and Domino SSO issues

3.13.3. Database Security
When looking into Database security, there are typically several questions Domino
administrators ask. Can anonymous users access the data? Is there a way to force SSL
access to this data? How do I control what page is displayed when the database is
opened? Can I prevent certain databases from being viewed with a browser? You will
now learn how to answer these questions.
Anonymous Access to Domino data
For additional protection, Domino has an additional level of security within the Access
Control List (ACL) of databases when accessing using a web based protocol such as
HTTP, DIIOP, POP3 or IMAP. Any Domino database located on a server running the
HTTP task should have an anonymous entry in the ACL. While you can disable
anonymous access at the server level as seen above in figure 2, it is best practice to add
an anonymous entry to the ACL of each database on the server. If anonymous is not
present in the ACL, then the Default access will be granted to anonymous users. Add
146
the anonymous entry is especially important on mail databases to prevent anonymous
users from accessing public calendar documents as many users allow or delegate read
access to their calendar to everyone. You can also set a maximum authority value when
accessing data from the internet. For example, if you have an application you want visible
from the web, but do not want anyone to edit the data from the web, you could set the
Maximum Internet name and password to reader. This means that when anyone access
the Domino database using one of the web based protocols, they will only be granted
reader access, even if they are specifically listed in the ACL of the database with greater
access. For mail files, the recommended Maximum Internet name and password setting
is editor.

The Domino Administrator client makes it easy to modify the ACL of many databases at
once. For example, to modify the ACL for all files in the mail directory, you can right click
on the folder and select Access Control Manage. The Manage Multiple ACLs
window will display. You can then see that at the top of the screen how many databases
you will be modifying. From there you can use the Add button and enter the value of
anonymous with an access level of No Access. Once added you should see anonymous
listed in the Apply these changes to all X databases as seen in figure 3.

147
You can then select the Advanced tab to modify the Maximum Internet name and
password setting to Editor as shown in figure 4.

Once you click OK, the client will then connection and modify the ACL for any selected
database that you as the administrator have authority to modify or all databases if using
full access administration authority. When finished, the client will tell you if the process
completed successfully or if there were any errors as seen in figure 5.

It is important to note that any database that already contains an entry for anonymous will
be listed in error. You can review the log or the status bar to see why an error occurred.
See figure 6.

If this happens to you, you can run the tool again this type changing the anonymous
148
entry. This way you can be certain that all databases are set to no access without
verifying each database.
SSL Access to Domino data
As an administrator you can force SSL access at the server level. For information on
configuring SSL refer to the technote How to set up SSL using a third-party Certificate
Authority (CA) or the Redbooks publication Domino Certification Authority and SSL
Certificates. However; most companies have public data that does not need to be
secured with SSL and private data that must be secured. In order to satisfy both
requirements you must force SSL connections at the database level. If you do want to
force SSL at the server level, you can do that by simply setting the TCP/IP port status to
Redirect to SSL as shown in figure 7.

To force SSL at the database level you need to set the database property Require SSL
connection. This is found on the basics tab as shown in figure 8. With this property
enabled, if a user attempts to access the database without SSL, they will be automatically
redirected to a secure connection. If you are planning to set this field on your mail files, it
is best to reference the users to the mail file with a secure connection using the iNotes
redirect application. Otherwise, the mail may not properly load for you due to the different
connection type between the mail file and the forms85.nsf. Forms85.nsf access is
needed in order for iNotes to display properly.

149

For more information about SSL refer to technote Frequently Asked Questions: Using
Secure Socket Layer (SSL) with Notes/Domino.
Additional Database Properties
There are other database properties that affect how your database can be accessed from
the web including:
Use JavaScript when generating pages
Don't allow URL open
Enable enhanced HTML generation
When opened in a browser / Database launch properties

3.13.4. File System Security
Do you have sensitive data or pieces of your web application within the data directory of
your Domino server? If so, the Domino server can access it and thus so can a user with a
browser. To be sure your files are safe, review the article Building Web applications in
Domino 6: Accessing and protecting the file system which is still technically accurate for
Domino 8.5.x environments.

150
3.14. Setting up a Redirection Application for Lotus
iNotes users
Do you currently use Lotus iNotes or are you considering doing so? If so, the topic of a
redirect application has probably come up. Domino provides a redirect application to
allow users to be routed to their mail by simply entering their user id and password. This
article describes the steps required to create an iNotes redirect database and make it the
default home page for the web site.

To help you understand the process of creating and implementing this application, you
will see a simple case study and Fictional Software Company A. The management team
at Company A has decided to roll out iNotes for all of their users. They have decided that
they would like users to be able to enter a URL of mail.companya.com and then the user
should be able to enter their user id and password and get directly into their mail. Here
are the steps the Domino administrator has taken to implement this request.
3.14.1. Create the iNotes Redirect Application
The first step is to create the iNotes redirect application. You can do this from the Lotus
Notes or Domino Administrator client (File Application New). When creating the
redirect application, you will first enter your desired Title and File name. You then will
then select to create the application based on the IBM Lotus iNotes Redirect template
as shown in figure 1.

151
3.14.2. Configure the iNotes Redirect Application
Once the redirect application has been created, you will be prompted with a Setup button
which you can click to get started with the configuration. From there you will be presented
with four options as shown in figure 2.

Server Settings

To get started with the configuration, click on Server Settings. The first choice to be
made is the Redirection Type. The value to choose here is dependent on your
environment. Use the table below to help you choose appropriate values for the
Redirection type and associated settings. In the example for Company A, they will be
using a Redirection type of Mail Server with TCP/IP domain of companya.com. Since
Company A does not use a proxy server or a specific folder for their mail files, those
settings will remain blank.
Scenario Settings
Multiple Servers running iNotes and users
should be redirected to mail file located on
their home mail server.
Redirection Type of Mail Server
Proper TCP/IP domain for the mail server
should be specified, for example
companya.com
Multiple Servers running iNotes, but all mail
files have been replicated on one server
located in a DMZ. All users should be
redirected to the copy of the mail file on the
DMZ server located in a folder called iNotes.
Redirection Type of Fixed
Server Name to use should be set to the
hostname of the DMZ server, for example
dmz.companya.com
Force path should be set to iNotes
Multiple mail servers and replica copies of the
mail files. Users should be redirected to the
replica copy of the mail file located on the
server that they are currently connected to.
Redirection Type of Dynamic
Single mail server environment. Users should
be redirected to their mail file on this server.
Any redirection type can be used in this
environment.
Redirect database exists on an application
server, but users should be redirected to the
appropriate mail server
Redirection Type of Mail Server
Proper domain for the mail server should
be specified, for example companya.com

The next decision to be made is SSL. At company A, the decision has been made to
force SSL during sign-in, but not use an SSL connection when accessing mail. Thus they
152
have set "Do you wish to force SSL for the entire session?" to No, but set "Do you wish to
force SSL only on authentication" to Yes. In the example provided here, the default SSL
port of 443 is being used so that setting has not been changed. Figure 3 shows all of the
server settings chosen by Company A. If you need to be sure SSL is used for
authentication and for iNotes access you can change the "Do you wish to force SSL for
the entire session" parameter to yes.

153
UI Settings
The next group of settings to be configured are the UI Settings. For the most part the UI
settings are self-explanatory. In the example shown in figure 4, Company A decided to
display their company logo on the standard redirect page. The redirect screen will be
shown for 4 seconds to allow the user to change the log in options. You will see an
example of the Enable Personal Options and Enable Login Options below.

154
When the user accesses the redirect database using their browser, they will see the
screen like the one seen in figure 5 for 4 seconds. Since Company A has chosen to
display their logo rather than the Lotus iNotes logo you can see that in the example.

If you choose Yes for the Enable Personal Options setting the Personal Optionslink will
be displayed as shown above in figure 5. Once the user selects this option, they will see
the screen shown in figure 6. Note that in order to allow the user change their log in
options, you need to set Yes to Enable Login Options, you must also have Yes for the
Enable Personal Options. In the Personal Options screen, there are two parameters
Alternative Mail File Display Name and Default View. The alternative mail file display
name can be used to enter another user name. In figure 6 you can see that User One
has logged in and selected to change the personal options. If User One has access to
User Two's mail, then User One could enter User Two in the Alternative Mail File Display
Name field and then User Two's mail file would be displayed rather than User One's mail
file after the redirection is complete. The Default View parameter can be used to switch
between the different iNotes formats or home pages. In this case User One is choosing to
use the Full mode. Options selected here are remembered and used the next time the
user logs in.

155
Ultra-light/Mobile Settings
The Ultra-light/Mobile Settings configuration within the iNotes redirect database is very
simple. You can determine whether or not your users can select the ultra-light mode and
determine which mobile devices should be automatically redirected to the ultra-light
interface. Within Company A they have rolled out iNotes ultra-light and thus it has been
enabled as shown in figure 7.

Application Setup
After you have successfully chosen your settings the last step is to access the Application
Setup option in the iNotes redirect application. Click on Click to Auto Set ACL Settings
to properly configure the ACL for your redirect application (reference figure 8).

Once the ACL has been properly configured you will get a message confirming the
changes as shown in figure 9.

You are now ready to begin using the iNotes redirect application.

3.14.3. Configuring the iNotes Redirect Application as the Default
Home Page
Once you have configured, you may wish to have the iNotes redirect application be the
default home page for your mail server. To do this, you will need to modify the default home
page setting in the server document for the server or in the internet site document (if you
use internet sites in your environment).
156
Setting the Default Home Page in a Server Document
If you are not using internet sites, you will need to set the default home page in the server
document. To access the server document in the Document Administrator client, go to
the Configuration tab. From this tab select Server All Server Documents. You can
then open the document for the appropriate server. To do this, you will access the
Internet Protocols... HTTP tab. Set the Home URL: to /<name of your redirect
application ?Open. For company A, the correct value would be
/iNotesredirect.nsf?Open as shown in figure 10.

Setting the Default Home Page in an Internet Sites Document

When using internet site documents, each site has its own home URL. Thus, the home
URL must be defined within the internet site document. To access the internet site
document in the Document Administrator client go to the Configuration tab. From this
tab select Web Internet Sites. Within the site document, access the Configuration tab
and define the Home URL. For example, set the home URL to /iNotesRedirect.nsf?Open

You have now seen how easy it is to implement the iNotes redirect application and set it
as the default home page for your server.

157
3.15. Securing Lotus iNotes
There are a number of security topics that are specific to iNotes, including Notes ID files,
browser cache management, encrypting offline databases, and encrypted mails
(S/MIME). This article addresses these topics and covers the key points for these topics.
3.15.1. iNotes and the Notes ID files
For authentication purposes an ID file is not required when using Lotus iNotes. However;
there are some functions that require an ID file including offline access to iNotes,
recalling messages, and sending and receiving encrypted mail (S/MIME). When you
register a user, you have the ability to have the ID file automatically stored in the user's
mail file. Starting at 8.5.1, the password in the stored ID file can be synchronized using
the ID Vault functionality of Domino. You can easily see if an ID file is stored within the
mail file by accessing the security preferences within iNotes as shown in figure 1.

For additional information on synchronizing passwords or enforcing a custom password
policy, refer to:
Can I synchronize a Notes ID password change with an imported Notes ID in a DWA mail file?
How to implement a Custom Password Policy for DWA Users

3.15.2. Active X Controls
ActiveX controls provide a way to create distributed applications that work over the
internet through the Internet Explorer browser. In Domino 8.5.x, there are several iNotes
functions that are implemented via an ActiveX control including: ability to drag and drop
attachments, ability to select multiple files when uploading and downloading files and
browser cache cleanup.

In order for an ActiveX control to install, the user must have Administrator or Power User
authority on their workstation. They must also allow the installation of ActiveX controls
within their browser. As an administrator you can control whether or not these controls
will be used by enabling or disabling them at the server level in the configuration
document as seen in figure 2 or for specific users in a mail policy.
158

3.15.3. Browser Cache Management
Browser cache manage is a way to define which temporary files stored on the PC during
iNotes access should be cleaned up after the user exits iNotes. For more information
refer to What is Browser Cache Management and how is it installed?

Frequently Asked Questions regarding Browser Cache Management:

Question: What happens when you log out of iNotes without the browser cache
management feature enabled?
Answer: The browser cache will be cleared when clicking logout, regardless if the
Browser Cache Management feature is enabled. The benefit of Browser Cache
Management is that the browser cache will be cleared without having to click logout. Just
closing the browser, for example, will clear the browser cache.

Question: Is there any way to force the user to restart their browser after automatically
installing browser cache management?
Answer: iNotes is a web based application and therefore must comply with the limitations
of the browser. iNotes does not have the ability to force the browser to close.
159

Question: What does it mean by clear history when the browser window is closed?
Answer: This refers to clearing the temporary files available in your web browser. The
files are deleted from the appropriate directory. You can set the cache scrubbing level to
remove all cache entries or only those related to the user's mail file. Since iNotes is a
web application, it has limitations on what it can and cannot do. When it clears the
browser cache, it deletes these files from the directory, the same as if you browsed to the
directory and deleted them via Windows explorer. It is important to note that other than
attachments, user's data is not stored on the hard drive, only design data to improve
iNotes performance.

Question: How does the attachment gets handle when you just read it and detach it?
How does it remove from hard-drive?
Answer: The same procedure applies: The file is deleted the same as if you browsed to
the location and deleted it manually.

3.15.4. Encrypting Offline Databases
One thing great about iNotes is that if your server is accessible from the internet, then
you can easily access your mail anywhere. Lotus iNotes also allow users to access their
mail offline (requires DOLS). When a user configures an offline subscription, a local copy
of their mail file is created. This can be a concern knowing that users could use a shared
computer. One way to minimize this concern is to force a user to encrypt their mail file
when going offline. This can be easily accomplished in the configuration document for
your server. In the example shown in figure 3, the Domino Administrator defined that
Medium encryption should be used and the user cannot change this setting.

160
3.15.5. S/MIME
Lotus iNotes fully supports sending and receiving encrypted messages using secure
MIME (S/MIME). There are 2 requirements for this functionality. First, the user must
access their mail via a secure connection (SSL). Secondly, a Lotus Notes id file must be
stored within the mail file. Sending a secure message within iNotes is very simple. If the
requirements have been met, the user can simply check the Encrypt option before
sending the message.

If the recipient of a secure message does not have an ID file stored in their mail file, a
warning message is displayed as shown in figure 5.

As you can see in the example, the error states that the body of the message is
encrypted. This is an important thing for you and your users to understand. The subject of
the message is not encrypted so all confidential information should be kept in the body of
the message.

161
Part 4. Tuning the Environment

4.1. Health Check
The purpose of a Health Check is to perform a reasonably deep analysis of the Domino
server configuration and underlying services it requires (such as hardware resource, disk
and network). A thorough health check aims to identify where best practices could be
implemented. It should also uncover any vulnerabilities that may compromise integrity,
availability or confidentiality of the information served by Domino.

This article provides Administrators with a checklist to help them get started with
performing a Health Check. Additionally, certain points are explorer in greater depth.

A health check compromises three key elements:
1. Inputs (Gathered by reviewing hardware and software configurations)
2. Analysis (A review of the inputs, in the context of your Domino environment)
3. Recommendations (Actions identified from the analysis phase)
4.1.1. High Level Checklist for Health Check
If you cover all these points, you will have performed a comprehensive health check:
1. Inspect Server Hardware
2. Inspect Server Configuration
3. Inspect Domino Mail Routing
4. Inspect SMTP Mail Routing
5. Inspect Domino Replication
6. Inspect Directory Services
7. Inspect Domino Clustering
8. Inspect Domino Security of Core Databases (names, admin4, events4)
9. Inspect Domino Server Topology
10. Inspect NOTES.INI files
11. Inspect Server Logs
12. Inspect Server Statistics
13. Inspect Server Documents
14. Inspect Connection Documents
15. Inspect Server Configuration Documents
16. Inspect Program Documents
17. Inspect Active Server Tasks
18. Report on Disaster Recovery Procedures
19. Report on Backup Procedures
20. Report on Number of Databases and Size
21. Report on Database Activity
22. Report on Agents and Runtime

162
4.1.2. Performing the Health Check
A health check is more than just looking at your environment. A good health check
requires that you document your findings to be reviewed with your team or management.
It is also a way to track your environment health. Throughout the rest of this document
you will see not just the steps to perform the check, but guidelines on what data your
report should include.
Environment Diagrams
In order to provide some context to the health check it is useful to include high-level
architecture diagrams. This will assist the reader to understand the key components that
make up the Domino environment.

Diagrams should succinctly describe the following information to the reader:
The number and physical location of Domino servers
IP addresses/host names
Domino clusters
The client types accessing the environment
The network environment
External access (from the internet)
Mail routing topology
Replication topology

For additional information refer to LINK to Doc topic
Current Client Environment
Use this section to get a picture of the user base that the environment serves. Describe
the approximate number of users and their access methods (Notes client, browser,
mobile device etc.). Person documents inside the Domino Directory often contain a list of
Notes client versions used to access the server if you are unsure which client types are in
use. This information is stored in the Administration tab. You can also enable license
tracking to give you an easier way of reviewing this information.
Domino Server Builds
This section of your health check is similar to an inventory. It lists the Domino servers in
the environment together with their role, host operating system and version (including any
hot fixes).

Take note of mixed version environments, especially within across members. Best
practice suggests that cluster members should all be at the same Domino version (e.g.
8.5.2).
Hardware

Your health check should include an overview of the current hardware being used.
Monitor key system resources over an extended period of time. You should aim to
monitor the systems for weeks or months. This will help to iron out any irregular variation.
Reporting on resource utilization has the following two benefits:
Ascertain a steady-state baseline of resource utilization
163
Help identify any pattern of spikes in resource utilization that can have an adverse
effect on the service.
Help plan when additional hardware or a server migration may be required

Operating tools such are useful. As a minimum, the following attributes should be
measured:
Total CPU utilization
Total Memory (RAM) utilization
Average Disk Queue Length
Network utilization
The tools used and the exact statistics will vary slightly between platforms. Below is an
example using Windows Performance Monitor (Perfmon.exe).

Figure 1 depicts a healthy server in terms of the resources outlined above. CPU
utilization is typically low with spikes not exceeding about 50% during busy periods.
There is plenty of free memory throughout the monitoring period. The average disk queue
length is typically under 2, except occasional spikes.

Figure 1: Healthy server

164
Figure 2 depicts a server that reaches its CPU capacity and has most of its available
memory utilized (up to 70%).

Figure 2: CPU capacity reaching peak

Figure 3 depicts a server where the average disk queue is very high, peaking at 80.

Figure 3: Average Disk Queue Length too high.
165
Disk Subsystem

The disk subsystem is a key component in the Domino server configuration. Various
factors such as the location of files, the RAID level and free space should not be
overlooked.
Disk Layout

Best practice suggests you should have separate disk volumes (separate spindles) for
the following components:
1. Operating System
2. Domino binaries (Program Code)
3. Domino data directory
4. DAOS Repository
5. Transaction Log drive
6. View Rebuild drive
7. Swap Drive
Note: On IBM i the operating system automatically manages your drives through its
single level storage capabilities. The operating system and all Domino components can
reside in the primary auxiliary storage pool (asp) with no negative performance side
affects. Using the primary asp is the default and recommended configuration for this
platform.
Free Space

As a rule of thumb, maintain at least 20% free disk space to reduce the amount of file
fragmentation. Performance usually degrades as a system runs out of disk space. If there
is no available space remaining this can lead to a server crashing or panic.
RAID Level

RAID at the OS/Software level is not recommended. Hardware-based RAID controllers
should be used in production Domino servers. There is always a balance to achieve
between the disk IO performance, redundancy and usable disk space gained from each
RAID level. Figure 4 lists the most appropriate RAID level for each Domino component.

Note: On IBM i/5, RAID level 5 or 6 is suitable for all volumes.
Domino Component Windows/Linux/AIX
Operating System RAID 1
Domino Binaries RAID 1
Domino Data RAID 10 or RAID 5
DAOS Repository RAID 5
Transaction Logs RAID 1 or RAID 10
View Rebuild Temporary Files RAID 1
166
O/S Swap drive RAID 1
Figure 4: RAID Levels

Furthermore it is best practice to use a separate RAID controller dedicated to the
transaction log drive.
View Rebuild Drive

Dedicate a RAID 1 array to store temporary files used to perform view index rebuilds.
This can improve performance especially during busy times when many users open their
Inbox for the first time. IBM Technote #1090462 documents the procedure in more detail.
Practice suggests that the rebuild drive is configured to use a dedicated RAID 1 array
composed of two dedicated solid-state disk drives or physical disk drives.

Note: For IBM i/5, the view rebuild directory may be in the primary ASP, but can be
located outside of the server's data directory for increased performance.
Transaction Logging

During a health check a review of transaction logging configuration should be done. In
most cases, Transaction Logging can safely be enabled in order to reap the benefits. An
excellent guide to Transaction Logging best practices is available in IBM Technote
#7009309.

Transaction Logging configuration should differ according to several factors. These
include:
Available Server Disk Layout
Available disk space
Domino Server Usage (Sametime server or Mail server etc.)
Type of backup strategy and backup tool availability
This topic is discussed further in the Transaction Logging section of this wiki.
Server NOTES.INI

To determine the health of your notes.ini consider the following:
The server NOTES.INI configuration file contains settings that can modify default
behavior and be used to tune the server. Care should be taken when modifying the
NOTES.INI. Always take a backup and document any changes in your change
management process.
Directly modifying the NOTES.INI file can lead to mistakes. An accidental or incorrect
change may cause Domino or Notes to run unpredictably. Therefore setting
NOTES.INI parameters in the server configuration document or using the set config
console command is safer.
A text comparison tool is useful to highlight differences in NOTES.INI files. Look for
consistency across server roles (e.g. mail, hub or application servers), and especially
across cluster members.
167
The IBM Lotus Notes and Domino wiki is a good place to get familiar with current
NOTES.INI parameters and their uses.
The NOTES.INI file tends to contain more obsolete parameters on servers that have
undergone upgrades from earlier Domino releases. NOTES.INI parameters typically
become obsolete because its function is superseded by a UI setting (in the server
configuration document, or server document itself).

An example is MAILCLUSTERFAILOVER=1. This parameter was superseded by a
setting in the Router/SMTP tab of the server Configuration document in Domino R5.
While the NOTES.INI parameter will not overwrite the setting in the Configuration
document, it can lead Administrators to think its set when its really not.
For NOTES.INI parameters obsolete in Domino 7, read IBM technote 1207338.
For NOTES.INI parameters obsolete in Domino 8, read IBM technote 1327806.
Redundant Tasks
The ServerTasks parameter specifies tasks that begin automatically when the Domino
server starts. These tasks consume memory and CPU so it is important to ensure only
tasks necessary to the server's role are included. An example of a redundant tasks is the
Rooms & Resource Manager (RNRMGR) running on a Sametime community server or
SMTP running on an administration server. As part of a good health check you should
ensure that all running tasks are still needed in the current environment.
Domino Configuration Tuner
In a nutshell, the DCT examines server configurations and compares them with an
extensive set of best practice rules. It allows administrators to quickly and easily analyze
an entire Domino domain and identify any parameters that are known to cause issues.
The DCT is explored in more depth in 4.2 Document Configuration Tuner (DCT) . As
part of any health check you should use the DCT tool and document the results and
recommended actions.
Database Size
The convenience and versatility of electronic mail makes it somewhat like an ever
growing file cabinet (or attic). Large mail files, especially ones that contain a large
number of documents, have a negative impact in several ways. For example:
They rapidly consume server disk space, especially when they are replicated to
multiple servers.
Database view indexes consume more disk space, consume more CPU time and
take longer to update.
The Inbox (and other folders) require more time to update and open.
Full-text indexes are larger and take more server resources to maintain.
Backups and restores take more time to complete.
Retaining old documents may violate document retention policies.
Having many large files open simultaneously can exhaust server resources,
especially on Windows.

For a discussion of large Domino mail files, see the paper entitled How Large Databases
Uniquely Affect IBM Lotus Domino Server Performance.
168
There are a variety of options available to Domino Administrators that help control the
size of user's mail files. Firstly there are database size quotas, and the ability to withhold
mail delivery from any mail file that is over its quota. Additionally, the size of individual
messages that can be delivered by the router can be limited. Other advanced database
properties discussed elsewhere (such as compression) can be effective at reducing the
overall database size.
These topics are covered in more detail in 2.2 Managing a User's Inbox.
For best performance, keep as few documents in the Inbox as possible. As a rule of
thumb, over 1000 documents is excessive. The Inbox Maintenance feature can be
enabled can be run periodically to move documents out of users' inbox and into another
folder.
Ports and Network configuration

As part of a health check you should review your network configuration including the
ports defined on your server. Here are a list of tips to consider and document as part of
your health check:
The Domino server Ports configuration is defined by NOTES.INI parameters. The
Ports parameter should contain all enabled ports defined on the server. The order is
also important as the first port listed denotes which should be used for cluster
communication.
Port compression allows network traffic to be compressed before being sent to its
destination. This may be either another Domino server, or a Lotus Notes client. Use
the Administration client (Server -> Status tab, Tools -> Ports -> Setup) to enable
or disable port compression. Do not enable port compression if your network
switches and routers already compress traffic automatically as you there is a CPU
overhead involved in compressing and decompressing the traffic.
Best practice suggests a dedicated NIC and dedicated high speed network for
Domino clustering to provide the fastest throughput. By using a dedicated network for
cluster replication you offload the cluster's probe and replication network traffic from
the user-access LAN, leaving more bandwidth for client communication with the
cluster servers. This also provides a level of redundancy in the event of a network
failure. For further information, read Configuring a Private LAN for a Domino Cluster.

Figure 6 is an extract from the server NOTES.INI of a cluster member that has the
Server_Cluster_Default_Port parameter set. With this configuration, the cluster
replication (clrepl) task only uses this defined port. If this port fails, the clrepl task does
not fail over to any other port.
Ports=TCPIP_CLU,TCPIP
TCPIP=TCP, 0, 15, 0,,32
SERVER_CLUSTER_DEFAULT_PORT=TCPIP_CLU
TCPIP_CLU_TCPIPADDRESS=0,10.172.99.134:1352
TCPIP_TCPIPADDRESS=0,10.172.99.100:1352
Figure 6: Cluster configuration extract from NOTES.INI
169
The Server_Cluster_Auxiliary_Ports NOTES.INI parameter will allow cluster
replication to fail over to the other available ports, even when using the
Server_Cluster_Default_Port= parameter. Its use is documented in IBM
technote#1259288. In this case, add the parameter like so:
Server_Cluster_Auxiliary_Ports=TCPIP
Verify that all servers are configured to use the same line speed and duplex options
as the switch to which they are attached to avoid potential network performance
issues.
Replication Topology

As part of your health check your replication topology should be reviewed and analyzed.
Consider the following:
If you are using a hub & spoke topology, best practices suggests initiating replication
from the hub. This is to maximize the amount of resource available on the spoke
servers to server client requests.
Best practice suggests the number of replicator tasks should equal to the number of
spoke servers with which the hub replicates. However, you should not exceed 20
replicators to avoid putting too much load on the server. If the server you intended to
replicate is not a hub server, the recommended number of replicators should equal
the number of processors on the server.
How often are you currently replicating? Is one replication event completing before
the next event begins?
How long does it take changes to be completely propogated in your environment. Is
this timing still acceptable or have business conditions or requirements changed?
Mail Routing

As part of your health check your mail routing topology should be reviewed and analyzed.
Consider the following:
Verify the number of mail boxes on each Domino server. This setting is found in the
Configuration document, Router/SMTP Basics tab. Follow the guidance in
Determining how many mail.box databases to place on a server to verify the optimum
number of mail boxes.
Is SPAM currently under control in your environment? Do you need to make changes
to your SMTP inbound controls or consider using a 3rd party service or product to
better manage SPAM in your environment.
Is dead mail accumulating in your mail boxes?
170
Server Security

In a complete health check security should also be considered. Here are a few tips. For a
complete list refer to 1.3 Security checklist.
The Security tab of the Server Document contains fields that control server access
and permissions. Organisations should have controls in place to ensure people are
given the minimum amount of access necessary to perform a task. For instance,
Database Administrators do not need Full Administrator Access to the Domino
Domain.
User and Server IDs can be attached to respective documents in the Domino
Directory, however best practice is to securely store them outside of the NAMES.NSF
and use features such as Certificate Authority and ID Vault.
Rather than grant default ACL access to the Domino Directory, Administrators can
set Default to No Access, then grant just the appropriate certifiers. For example,
*/ORG to Reader.
Directory Assistance

As part of a health check you should review your directory architecture. For information
on using multiple directories, see 3.4 Multiple Directories.

Directory assistance is a feature that enables a server to look up information in a
directory other than a local primary Domino Directory (NAMES.NSF). You can configure
directory assistance to use either a remote LDAP or another Domino Directory.
Secondary Domino Directory referrals should be configured to use a local replica to the
server for best performance. Administrators should create replicas of additional Domino
Directories referred, on all servers where Directory Assistance is enabled.

Also consider the following:
Are additional secondary directories required?
Are all directories currently being used still needed?
Should a mobile directory catalog or extended directory catalog be created?
Is a mobile directory catalog being used on a server instead of the recommended
extended directory catalog architecture?
Person Documents
As part of your health check, a review of the person documents is recommended. For
example, field validation for the Person document does not specify that the Internet
Address field be complete. Mail routing from external senders can still occur if an internet
address is stored in the User Name field. To help prevent against incorrect mail routing,
all members with a Domino mail box should contain a valid external email address in the
internet address field.
Policies
Policies are a powerful tool that enables Administrators to control many user's client
settings. These can be used to simply set client options (such as spell-check before
sending e-mails). They can also be used to enforce company policy (such as e-mail
message disclaimers).
171

As part of your health check review which policies are in force. Also check policy rules for
any contradictory policies that might be applied to users. It is good practice to have at
least a Desktop, Security and Registration policy:
Desktop Policy to help standardize the Notes client. Benefits include a reduction in
support calls and easier troubleshooting.
Security Policy to enforce password for compliance in-line with corporate instructions.
Registration Policy to standardize creation of new users and their accounts. For
example, ensure all new users have Editor access to their mail file (rather than
Manager).
The topic of Policies is discussed in more detail in this wiki. See 2.3 Policies.
Backup and Restore Procedures
As part of your health check you should review and document, if it has not been done
already, your backup and restore procedures.
A clearly defined backup procedure is vital for any business. The backup schedule for
Domino environments will depend on whether Archived Transaction Logging is
enabled. If the Transaction Logging type is circular logging, or is disabled, then daily
full backups of the Domino data (NSF / NTF) is most likely required.
If the transaction logs are backed up with a third-party backup tool, then the schedule
must accommodate this. Transaction logs should be backed up frequently. Each
evening an incremental backup captures the Domino data. A full backup should still
be taken once a week.
Equally important as the backup procedure is the restore procedure. Whereas the
backups tend to be automated, a restore is usually performed by the technical
operations group or Domino Administrators. It is therefore useful to perform periodic
dry-runs, to ensure databases are restored as expected and the procedure can be
performed smoothly when the business demands it.
4.2. Document Configuration Tuner (DCT)
The Domino Configuration Tuner (DCT) is just one of many tools that are included with
the Domino 8.5.x server and Administration client. It can also be downloaded for free
from this site: http://www-01.ibm.com/support/docview.wss?rs=463&uid=swg24019358

DCT provides easy-to-use self-service configuration analysis by Domino Administrators.
Its goals are to identify server document and NOTES.INI configuration parameters that
could be modified for more robust and better performance of Domino servers.

DCT will analyze your configuration against a set of best practice rules identified by IBM.
Furthermore, it warns of worst practice configurations if discovered. Configuration
settings are flagged when their values are know to cause problems based on Technotes,
Redbooks/Redpapers, internal knowledge from IBM development and support and prior
customer experience. Out-of-range and unexpected values are reported so that
undefined behavior can be prevented. Links to supporting to documentation are also
provided.

172

DCT rules are tuned for Lotus Domino server 7.0 and later (although it can analyze any
version of Domino) and requires the Lotus Notes client, standard or basic, version 8 and
later. Inside the DCT tool is a button that installs the latest updates to the DCT template
and downloads the latest rules. Rules are typically updated each month and up-to-date
details of the current additions can be found on the Tuner Blog.

No configuration changes to the production deployment are required to perform a DCT
evaluation. A full analysis exhibits a negligible load on the environment, even in business
hours. Therefore analysis can be performed at any time.

Most of the rules that are evaluated have a wiki post in the Notes/Domino wiki and DCT
will point to that wiki posting in your report. It is recommended for you to take advantage
of DCT to feedback your results, especially if your experience does not match the
recommendation in the findings. The Notes/Domino wiki entries provide a good place to
do so.

For further reading, visit the Domino Configuration Tuner entry in the Notes/Domino wiki.

4.3. Establishing a Performance Baseline
Having a set of historical baseline data gives Administrators a reference point to measure
against when changes are made to the environment. Establishing the baseline also gives
Administrators an opportunity to identify obvious performance bottlenecks or trends. For
example, the CPU may reach peak levels every Monday morning or disk queue lengths
may exceed normal operating boundaries during the same periods each week.

To help capture accurate baseline historical data, follow this set of best practices:
Capture a complete view of the infrastructures performance: Kernel statistics (such
as CPU, memory and disk utilization).
173
Periodically capture the information to get long term trend analysis. Do not rely just
on a quick snapshot of a system before a tuning event take place.
It should be representative of normal system loads. Do not record or take
inconsideration of data taken during extra-ordinary events (periods of unusually quiet
activity, such as the New Year holiday period).
Do not include data from periods with unusual events (such as system failures or
high usage volumes).
Record at a set period of time (For example, at least once a week, and preferably at
least once a month).
Include descriptions of what components or services were running on individual
servers (ServerTasks, Backup or Anti-virus facilities) at the time of capturing the data.
After every major tuning changes or system upgrades (when the system is stabilized),
take a new set of baseline statistics.
4.3.1. Recommended Metrics
The following metrics provide a basis for establishing baseline statistics. With experience,
you will find other metrics useful or more applicable for the type of tuning performed.
Perfmon
Memory.Available MBytes
Processor.% Processor Time
LogicalDisk Avg. Disk Seconds Read
LogicalDisk Avg. Disk Seconds Write
LogicalDisk.Avg. Queue Length.
- o/s drive
- Notesdata
- tx log
- DAOS
- view rebuild temp. drive
- swap
Events4 / Statrep
Server.AvailabilityIndex
Server.Users
Replica.Cluster.SecondsOnQueue.Avg
Replica.Cluster.WorkQueueDepth.Avg
Note. The Replica.Cluster statistics are only relevant if the server being monitored is a
member of a Domino cluster. They give an indication of the ability of the cluster to
maintain synchronized replicas.

For a comprehensive guide to Domino statistics and their definition, read Lotus Domino
Statistics.
174
4.3.2. Collecting Domino Statistics
Administrators can use the COLLECT and EVENT server tasks to monitor and collect
server statistics in entire Domino domains. The procedure is well documented in the
Statistics and the Domino System, IBM Lotus Domino and Notes InfoCenter article. The
sample interval should be no lower than 15 minutes, although Administrators may find 1
hour suitable.

Statistic collection can be performed on any Domino server. A best practice is to
configure statistic collection on an Administration server and remotely collect statistics
from other servers. The topic of Centralized Vs Stand-alone statistics collection is
discussed in IBM technote 1092952.
4.3.3. Reporting Database
Figure 1 shows an example of a customized Monitoring Reporting database
(STATREP.NSF). It contains a document for each sample according to the frequency set
in the events4.nsf, statistics profile. Add or remove columns, and choose from all
available statistics. These statistics can easily be exported as a comma separate volume
file, so graphs can be produced.

Figure 1: Monitoring results in statrep.nsf

175
4.4. Domino Tuning Tips (all platforms)
The following tips will help Administrators to optimize their Notes/Domino environment. It
is important for Administrators to understand that any configuration change should be
closely monitored for its effects. Make single changes at a time to ensure adverse effects
are easy to spot and changes can be backed-out.
4.4.1. View Index Updates
It is possible to tune the Domino update task to take advantage of the additional CPU
resource of multi-core hardware. There are a several NOTES.INI parameters that you
may use:
Update Task (View Index Updates)

If the systems resources allow it, additional Update tasks may be started on the server,
by adding the parameter, Updaters=[number], to the NOTES.INI. As a rule of thumb, do
not exceed a maximum of one Update task per CPU. For example, to enable two Update
tasks, apply this parameter:
Updaters=2
Full Text Index Updates

A separate thread can be created solely for full text updates. This separate thread will
only be responsible for updating full text indexes, so view updates will continue to occur
even if a large full text index is being rebuilt. Consider applying this NOTES.INI
parameter if many full text indexes are present on a Domino server and sufficient CPU
resource remain:
UPDATE_FULLTEXT_THREAD=1
Maintaining view indexes

The NOTES.INI parameter DEFAULT_INDEX_LIFETIME_DAYS=[number of days]
allows administrators to set a default lifetime for database view indexes if none was
selected by the database designer in the Design View Attributes dialog box.

The default value for this parameter is 45. Setting this to a lower value can have two
benefits:
Save Disk Space
Reduce amount of work Update task must perform
Creating view indexes is disk intensive and requires CPU and memory resource, so it is
important for Administrators and designers to strike a balance. It is not recommended to
set this value to lower than 14 (two weeks).
176
4.4.2. Disable Transaction Logging For Certain Databases
Turning transaction logging may impact your enabled DAOS features relative to mail. In
most cases, disabling Transaction Logging is not recommended because you lose the
benefits of fast server restart. Disabling transaction logging on a database will cause
Fixup to run on that database, creating the potential for slow restart. Furthermore, you will
not be able to perform incremental backups using the transaction logs for that database.
For some databases, however, it might not be necessary to enable transaction logging.
This is because it generates additional transaction log traffic on the disk and causes the
transaction log drive to fill up quicker. Examples of databases that are suitable for
disabling of transaction logs are:
Mail boxes (mail.box) - Do not disable Transactional Logging on Mail.boxes if you are
running DAOS.
Server Log (log.nsf)
Monitoring Results (statrep.nsf)
Statistics Mail-in (statmail.nsf)
Web Administration (webadmin.nsf)
Schedule (clubusy.nsf or busytime.nsf)
Cluster Database Directory (cldbdir.nsf)

Table 1 highlights NOTES.INI parameters to force new instances of the certain system
databases created on server startup to have transaction logging disabled. They do not
disable transaction logging on existing databases, or if they are created manually.

NOTES.INI Parameter System Database
MailBoxDisableTXNLogging=1 Mail.box (See note below)
Log_DisableTXNLogging=1 Log.nsf
Schedule_DisableTXNLogging=1 Clubusy.nsf or busytime.nsf
Table 1: NOTES.INI parameters to disable database transaction logging.
Note: Do not disable Transactional Logging on Mail.boxes if you are running DAOS.
4.4.3. Replication
If you have more than one Domino server in your environment then you will need to set
up replication. The following tips should help optimize the time and resource
requirements to replication data around your environment.
Multiple Replicator Tasks
The database replicator (replica) task on a server is responsible for handling scheduled
replication requests, as configured in server connection documents. The Cluster
Replicator (clrepl) task is event driven and responds to changes in a database. These
changes are then replicated to other cluster members.

Each replicator task only replicates with one server at a time, therefore the first replication
must complete before the next one can start. The number of concurrent replicator tasks
running can be set with the following NOTES.INI parameter:
177
Replicators=[number]
In environments where more than two servers participate in the cluster, additional clrepl
tasks can be enabled. The number of concurrent clrepl tasks running can be set with
NOTES.INI parameter:
Cluster_Replicators=[number]
Administrators should monitor Replica.Cluster Domino statistics. For instance, the
WorkQueueDepth Domino statistic indicates the number of changes waiting to be
replicated to cluster members. If it is continually increasing, enable additional clrepl tasks.
The following statistics give indication on the current, average and peak Cluster work
queue depth:
Replica.Cluster.WorkQueueDepth
Replica.Cluster.WorkQueueDepth.Avg
Replica.Cluster.WorkQueueDepth.Max
Replication Triangulation
When a client or server replicates with a remote server, it keeps a log of the name of the
remote server and the time and date in the Replication History. Domino uses the
replication history to determine which documents to scan for changes during the next
replication. The purpose of Replication Triangulation is to make each server aware of
every other server which maintains a replica of the same database, and which has had a
successful replication.

In a large environment (hundreds of servers in a domain), the number of replication
history events to maintain can cause a significant performance impact to the Domino
server. Maintaining replication triangulation history for databases which exist on hundreds
or thousands of servers is too expensive. This can manifest itself in the form of increased
CPU activity for the replica task.

You can disable replication triangulation with the following server NOTES.INI parameters
(available in Domino 7.0 and later).
NSF_REPLHIST_NO_TRI=1
REPL_NO_WS_TRI_HIST=1
REPL_NO_REMOTE_TRI_HIST=1
For local replicas, use Notes Client NOTES.INI parameters:
NSF_REPLHIST_NO_TRI=1 [This will prevent existing triangulated entries from
being read]
REPL_NO_WS_TRI_HIST=1 [This will prevent new triangulated entries from being
written]
Note: After setting the NOTES.INI parameters, the replication history must be purged
from each replica of the databases affected.

Further information is available in IBM technote 1270104.
178
4.4.4. Internal Caches

In Domino, caches are used to store frequent lookups in memory, to prevent the data
being continually read from disk. Since the mechanical hard-disk is commonly the
slowest component of the server, caches help improve performance and make a more
responsive user experience. The following sections provide Administrators with tips to
ensure cache sizes are properly set.
NLCache
NLCache is the name lookup cache. The default is 16MB before Domino 8.5.2. Beginning
in Domino 8.5.2, the default is 64MB. It can be increased as needed up to 4GB.

To determine if you need to increase the NLCache, use the show stat database or show
NLcache console command. For example:
Database.NAMELookupCacheCacheSize = 16,447,205 (current size of cache)
Database.NAMELookupCacheLookups = 1,879,903
Database.NAMELookupCacheMaxSize = 16,777,216 (default maximum size)
Database.NAMELookupCacheMisses = 1,362,746

A relatively high number of misses compared to lookups indicate that you should
increase the maximum allowed cache size. Administrators should increase the size of the
NLCache in increments, so try doubling the default to begin with, if necessary. Modify
using the ini parameter:
NLCache_Size= For example, set 67108864, which sets NLCache_Size to 64MB.
Group Cache
When the server needs to lookup the members of a group (For example, in the event of
an authentication request) it first checks the Group Cache. It will store results in the
Group Cache to optimize performance. Groups are invalidated during updates or when
cache is full. The default size is 4MB and it can be increased to 15MB.

If the cache needs to rebuild frequently because not enough data can be cached, this can
slow down group lookups. Therefore, frequent or very large group updates can slow
down server.

Verify GroupCache statistics with Show stat net
NET.GroupCache.Hits = 155
NET.GroupCache.Misses = 10
NET.GroupCache.NumEntries = 9
NET.GroupCache.Size = 65,406
NET.GroupCache.Used = 2,716

NET.GroupCache.Misses indicates the number of times a group was not found in the
cache and so had to be read from disk. Administrators should increase the size of the
Group Cache in increments until the number of misses reduces. Modify size with the
NOTES.INI parameter:
Group_Cache_Size= For example set 15360 , which sets Group_Cache_Size to
15MB.

179
Unified Buffer Manager (UBM)

The UBM (formerly referred to as the NSF Buffer Pool) is used to increase performance
by caching disk I/O requests to all databases on a server. The UBM is typically the
largest contribution of shared memory usage on the server.

In earlier releases of Domino, the maximum size of the UMB was automatically
calculated as 3/8th's of the physical RAM in a server. In a server with many gigabytes of
RAM, this could result in an unacceptably high UMB upper-limit. Consider a server with
4GB RAM installed:

Max UMB Size = (4 x 3/8) = 1.5GB.
If the UMB were to grow to its maximum size of 1.5GB, there is a risk that shared
memory may exceed process limits and result in a server instability.
Domino 8 now enforces a maximum UBM size of 512 MB default on Windows32, Linux,
AIX, Solaris, i5/OS and z/OS platforms. On 64-bit Domino, the maximum UBM size is
1024 MB. Therefore, there is no need to take action to properly size the UBM in Domino
8. For additional details on this new behavior, see IBM technote #1268988.
For recommendations on setting NSF_BUFFER_POOL_SIZE_MB, read IBM technote
1286171.
NSF_DbCache_MaxEntries
The NSF_DbCache_MaxEntries determines the number of databases that a server can
hold in its database cache at one time. If your server has sufficient memory, you can
improve the performance of the server by increasing the number of databases that Lotus
Domino can cache in memory at one time. The default value is 25 or the
NSF_Buffer_Pool_Size divided by 300 KB, whichever value is greater.

You should monitor the Database.DbCache.Hits statistic on your server. This indicates
the number of times a database open request was satisfied by finding the database in
cache. A high value indicates database cache is working effectively. If the ratio of
Database.DbCache.Hits to InitialDbOpen is low, you might consider increasing
NSF_DbCache_Maxentries.

For detail information on how to set the number of databases cached simultaneously in
Lotus Domino, read IBM technote 1279893.
NSF Monitor Pool
The Monitor Pool caches event monitors which include server and client mail rules. If set
too low the Monitor Pool can become full. Side effects include causing new calendar
notices not to display in the Calendar Mini View.

The default pool size is 40 MB. The maximum value for this pool is 256/512 MB.

To see the current values review stats
Database.MonitorPool.Event.Used = 52309
Database.MonitorPool.Monitors.Used = 1184
Database.MonitorPool.Size = 41943040

Check the LOG.NSF for the error Insufficient memory NSF monitor pool is full to know
if you should increase this pool for the Domino server. Many customers find a value
180
between 80 and 100 MB to work best. Increase this value if you are approaching the
maximum by using the NOTES.INI setting:
NSF_MONITOR_POOL_SIZE_MB=
4.4.5. Multiple Mail Boxes
Additional mailboxes are required on a server only when the amount of mail traffic
prevents the router from being able to effectively access the mail box files and access
conflicts are encountered. Access conflicts occur when other threads or processes lock
the mailbox, while another entry tries to access the mailbox and is denied.

The typical number of mail boxes on a mail server ranges from one to four, depending on
the volume of mail. IBM technote 1148438 provides Administrators with the procedure to
determine whether additional mailboxes are required on busy servers. If you feel you are
in need of another mailbox. Change the server configuration to 2 mail boxes.
4.4.6. Tips for Server Based Mail Rules
Keep the number to the absolute minimum, based on business requirements. Rules
require processing by the mail router each time it delivers mail. Therefore the fewer rules,
the fewer server resources are required.
Place rules most likely to be triggered at the top of list and use the Stop processing
further mail rules option.
Searching the memo body is CPU intensive and will cause and increase in the CPU used
by the SMTP task

Use the following NOTES.INI parameter to cap the number of mail rules per user's
mailfile or server mail.box:
MailMaxFilters=<1 to 100>
This IBM Case study provides Administrators with an indication to the impact mail rules
have on a server.
4.4.7. Tuning User Sessions
There are two NOTES.INI parameters that define how many client user sessions a server
can have concurrently open and also how long a session remains open. Keeping
sessions open requires memory on the server, whereas opening a new session
consumes a small amount of CPU resource. Therefore a balance must be achieved.
Server_MaxSessions=[number] (Maximum number of concurrent open sessions)
Server_Session_Timeout=[number minutes] (maximum duration of idle activity before
the Domino server terminates the session)

A timeout value of 30-45 minutes is recommended for most customers. For more
information read IBM technote 1089879.
181
4.4.8. Domino Configuration Tuner (DCT)
The Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration
analysis for more robust installations and better performance of Domino servers. Make
sure you are up-to-date with suggested changes from Lotus Domino by periodically
updating DCT. For more information, read "Domino Configuration Tuner (DCT) provides
easy-to-use self-service configuration" - IBM technote 4019358.

4.5. Tuning for Virtualized Environments
As organizations continue to try to optimize and leverage their hardware and
infrastructure environments, virtualization is becoming a larger footprint for many mail
and application server deployments. Virtualization can be one method of reducing TCO
(total cost of ownership). This article provides some best practices and guidance for
deploying Domino servers on a virtualized VMware architecture.

It is worthwhile pointing out that by virtualization alone, it is unlikely that a TCO reduction
will be achieved. This is because the number of Domino servers remains the same and
so does the Administration effort required. Furthermore there is another (host) operating
system to tune and maintain. Organizations may find TCO reductions can be realized
more effectively by implementing new Domino features, such as DAOS and ID Vault, or
Domino server consolidation (hosting more users on the same server).
4.5.1. The Pros and Cons of Virtualization
The potential benefits of deploying Domino in a virtualized environment include:
Organizations can reduce the physical footprint required for supporting a server
consolidation effort. You can run two mail servers on one physical server.
Organizations can leverage a higher amount of a server's total resources.
Migrating from physical hardware to virtualized hardware has little impact on the
users. Whereas migrating users to consolidated Domino servers can require from
template changes to hard-coded server references, and/or changes to users'
desktop configuration.
From a disaster recovery perspective, it can allow for an image to be brought up
on another VMWare server very quickly and communicate with the Domino Data,
DAOS and Transaction logging storage as if it were on the same physical server.
These potential benefits, however, do come with costs. Organizations must consider the
following facts:
There is a hardware resource overhead to provide the virtualization.
More Domino server virtual machines (VMs) typically mean higher disk and
network I/O, channelled to a single physical server.
Some configurations can lead to dynamic guest resource allocation. This can
lead to resources being shared across VMs.

182
4.5.2. Static Resources
A key to vitalizing your Domino servers is to ensure that the architecture within the
virtualized systems is static and dedicated. For example Disk I/O for Transaction logs,
DAOS, users mail files and networking functions should all be dedicated for each
Domino server. This is in order to optimize the I/O performance channels and minimize
the impact that the virtualization may have any virtualized node (or guest).

Ensure hardware resources are isolated and dedicated to the VM guests. Do not use
VMware configurations such as memory over commit, ballooning or dynamic VM
balancing as this can lead to stability / performance related problems for the Domino
application.
4.5.3. Best Practices for Guest VM
The following points are designed to be templates for a starting point in sizing Domino
mail servers.
Virtual CPUs
The number of vCPUs per VM depends on the number of users to be supported.
Try not to over-provision vCPUs of Domino guest. 4 CPUs should be fine in most
cases.
Virtual RAM
RAM for Domino allows it to cache more data. The operating system does I/O caching
too, which improves the overall performance. IBM Software Services for Lotus
recommends 4GB of RAM per Domino Server.
Improved memory limits in 64-bit OS helps cache more data, and thus avoid disk
IO. Reduces response times, and hence increasing the number of users.
Increase VM memory when running in 64-bit guest OS. Recommend each VM to
have 8 GB allocated for guest OS and Domino Application.services.
Storage

Storage architecture may be a bottleneck due to the number of servers and volumes per
LUN. To optimize the storage architecture, do not use shared LUNs. This means each
volume (such as notesdata or transaction log drive) is mapped to one set of spindles and
are unique to each Domino server.
Use Fiber Channel disks when ever possible, 4Gb or better is recommended. 2Gb is
supported but has limited bandwidth for Domino Mail servers.

Align partitions
Use VMFS and use Virtual Center to create partitions
Use separate, dedicated LUNs for OS/Domino, data and transaction logs
Separate the IO at physical disk level, not simply logical LUNs
183
Make sure these LUNs have enough spindles to support the IO demands
Fewer spindles or too many VMDK files on single VMFS LUN can substantially
increase disk IO latencies
RAID configuration
Best performance using RAID 1+0 for Data, RAID 0 for Log
Raid 5 can cause write queues to build up on slower Storage solutions i.e. iSCSI,
Hardware Storage or software-based RAIDs.
Networking Configurations

To optimize the network access, do not share physical network interfaces controllers
(NIC) with different Domino servers. Use dedicated NICs based on the network traffic:
Use separate NICs for mail and cluster replication traffic.
Use Enhanced VMXNET 2 or 3 driver or higher with TSO and Jumbo Frames
support.
Use NIC Teaming & VLAN Trunking if available - Note, Network teaming not
always the best way.
Note: Co-located VMs outperform physical 1Gbps network speed
VM Time Synchronization
Use VMware Tools time synchronization within the virtual machine
Enable NTP daemon to sync with external NTP source (using vSphere client)
Disable OS Time Service
Windows: w32time service
Linux: NTP daemon
Resource Reservation

Do not set limits for Domino resources. As the Domino servers are virtualized, they
should be done as Static resources for each Domino Mail Server.
Ballooning

Never use Ballooning for Domino servers. Domino allocates and uses the memory on
startup based on what it sees. This means Domino servers will build cache pools, based
on the total memory physically allocation. If you use ballooning you can remove or
compromise these configurations and cause Domino to slowdown, hang or crash.
In production environments ballooning should always be 0 for optimal performance.

184
4.6. Domino on Windows Tips
This article provides tips for Administrators running Windows-based Domino servers.
4.6.1. System Page Pool
Large environments with hundreds of gigabytes of Domino data can be affected by a
common issue with the Windows.

Domino maintains a database cache to improve performance by keeping recently used
files open for quicker access. These caches are stored in the Windows page pool and
may still remain in the page pool even if the database is closed by the client. If many
large databases are opened, it is possible to exceed the page pool limit and cause
problems for the Domino server. Read IBM technote 1093511 for more information.

To try and alleviate the issue on Windows 2003 (32-bit) servers, perform the following
steps to modify two(2) registry keys(these steps were taken from Microsoft document
Q312362).

1. Start Registry Editor (Regedt32.exe)
2. Locate and then click the following key in the registry:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session
Manager\Memory Management
3. On the Edit menu, click Add Value, and then add the following registry value:

Value name: PoolUsageMaximum
Data type: REG_DWORD
Base: Decimal
Value data: 60

Setting the value at 60 informs the Memory Manager to start the trimming process at
60 percent of PagedPoolMax rather than the default setting of 80 percent. If a
threshold of 60 percent is not enough to handle spikes in activity, reduce this setting
to 50 percent or 40 percent.

Value name: PagedPoolSize
Data type: REG_DWORD
Base: Hex
Value data: 0xFFFFFFFF

Setting PagedPoolSize to 0xFFFFFFFF allocates the maximum paged pool in lieu of
other resources to the computer (~500 MB).

Caution: The 0xFFFFFFFF PagedPoolSize setting is not recommended for use on
32-bit Windows Server 2003-based computers that have 64GB of RAM. This will
potentially bring the Free System PTE entry down and can cause continuous reboot
of the computer. For this configuration, carefully choose a value based on the
requirements and available resources.
185
4.6.2. Other Tuning Tips for Windows Servers
The Domino Server Performance Troubleshooting Cookbook is focussed on Windows-
based Domino servers and provides further reading for Administrators.
4.6.3. Additional references
The Knowledge Collection: Hardware or Operating System error - Insufficient system
resources on the Domino Wiki goes into detail about Windows Paged Pool issues. It
discusses the areas we mentioned as well as others.

There are also Windows 2008 specific articles that describe common performance
problems if you are not running with the fixes:
IBM technote 144982
"Windows 2008 64-bit can cause a significant CPU increase and I/O degradation
when Domino opens many databases"
IBM technote 1427348
"Windows 2008 64-bit hangs/slows dramatically when Domino server is shut down"
4.7. Domino on Linux Tips
Domino for AIX and for Linux share some characteristics that allow some consistency in the
approach to tuning on these platforms.
4.7.1. Monitoring Server Resources
4.3 Establishing a Performance Baseline already discusses the importance of establishing
the baseline prior to undertaking tuning changes. The nmon tool is designed to monitoring
and analyzing performance data, such as CPU and memory utilization, disk I/O metrics and
kernel statistics. The nmon performance tool can capture data to a text file for later analysis
and graphing for reports. The output is in a spreadsheet format (.csv).
4.7.2. Operating System Limits
The default setting for maximum number of open files allowed is usually set too low. If the
value set for this parameter is too low, errors might occur when opening files or
establishing connections. Since this value limits the number of file descriptors that a
server process might open, a value that is too low prevents optimum performance. Check
/etc/security/limits.conf and append the nofile entry as follows:
notes soft nofile 49152
notes hard nofile 49152
186
Note: In the above example, notes is the user account that Domino runs under. Set this
according to your environment.

Similar to open files, the maximum number of processes/threads can be set too
low. The nproc attribute represents the maximum number of processes each
user can create. Check /etc/security/limits.conf and append the nproc entry as
follows:
notes soft nproc 12500
notes hard nproc 12500

Note: In the above example, notes is the user account that Domino runs under. Set this
according to your environment.
4.7.3. Linux Services
(Specific to Red Hat Enterprise Linux): The updatedb command updates the locate
database. This process is very disk intensive and so should only be run out of business
hours. Verify that the mlocate.cron script (typically /etc/cron.daily/mlocate.cron) runs out
of normal business hours.
Disable these services to prevent port conflicts, if enabled on the Domino server.
httpd/apache (HTTP)
sendmail (SMTP)
4.7.4. TuneKrnl
Domino for Linux comes with a tool 'tunekrnl' that runs automatically when Domino starts
to set dynamic kernel parameters. Allow the Domino server to automatically calibrate
kernel parameters.

4.7.5. Troubleshooting and Debug Tips
On Linux, ensure you have installed the latest version of gdb (GNU Project
Debugger).
On AIX, ensure you have installed the latest version of dbx (the AIX standard
command-line debugger).
On AIX, ensure you have installed the latest version of procstack (Anoth AIX
debugger. Prints the hexadecimal addresses and symbolic names for all the threads
in the process).
Note: On AIX, Domino may use both dbx and procstack depending on the situation. Both
should be up to date versions.
187
4.7.6. Disabling concurrent I/O and direct I/O on Domino servers
on AIX
The CIO feature is not supported for use with Domino servers, do not enable this option
on file systems that Domino accesses. For further information read this Infocenter article.

4.7.7. Tuning Java for Domino on AIX
The following IBM technote 1451336, contains guidance on preventing excessive CPU
usage by Agent Manager threads.
4.7.8. Perfpmr for AIX
Perfpmr is a tool for AIX that gathers performance specific data from your system. It is a
shell script that runs other scripts that execute various tools to get information on system
performance. It is useful to review the information Perfpmr collected to troubleshoot your
performance issue or have IBM Support to help debug your system. You can run it when
you have performance problems, during the normal period, or before and after system
configuration changes or fixpack upgrades.
For more information on Perfpmr and a list of useful AIX tools, read Performance
Monitoring Tips and Techniques

4.8. Domino on IBM i Tips
4.8.1. Overview
Getting started with running Domino on IBM i? Here are some things you should be
aware of when running Domino on IBM i
On IBM i every Domino installation is a partitioned install. The Domino product code
is always stored separately from the Domino server data. You will find the Domino
product code located in the /QIBM/ProdData/Lotus/DominoXXX directory where XXX
represents your release, for example /QIBM/ProdData/Lotus/Domino852.
Domino on IBM i supports running multiple releases. For example, when planning
your upgrade you could upgrade your test Domino partitioned server (dpar) to 8.5.2
while leaving your primary dpar at 8.5.1. For more information about multi-versioning
refer to an old, but still relevant, IBM Redbooks publication Lotus Domino 6 Multi-
Versioning Support on the IBM eServer iSeries Server.
Domino installs and upgrades can be separate steps on IBM i. This means you can
install the product during the business day and then upgrade the server at a later
time. This will reduce the outage window required for the upgrade. The command
then used to upgrade the server is UPDDOMSVR. For more information on the
UPDDOMSVR command refer to UPDDOMSVR InfoCenter Topic.
All Domino data must be owned by the QNOTES user profile for proper operation.
This is particularly important for all objects in the Domino server data directory. To
188
update the owner on an IFS file use the CHGOWN command, to update the owner of
native objects use the CHGOBJOWN command.
IBM i is an EBCIDIC system. To prevent performance problems when transferring
data between platforms always use binary FTP - in other words, do not drag and drop
Domino databases from a Windows environment. For information on using FTP refer
to the technote Using FTP to transfer files to/from the Domino Data directory on IBM
i.
IBM i uses a different file locking mechanism than Windows. Thus, to prevent file
access and data corruption issues, do not map a network drive to a Domino server's
data directory.
There are graphical user interfaces available to use with the IBM i operating system
including System i Navigator and IBM Systems Director. If you want to learn about
the native CL commands most used by a Domino administrator refer to Multimedia:
Introduction to IBM i CL commands for Domino administrators.
For information on backup and recovery specific to IBM i refer to Backup and
recovery strategies education modules available via IBM Support Assistant.
Be aware of network changes or problems. Slow response in your network will cause
delays and errors for your users. Also, Domino must have access to a functioning
DNS server at all times in order to route mail. If the DNS server IP address changes
in your environment you must also update the DNS server entry within IBM i using
the CHGTCPDMN command.

4.8.2. Performance
When getting started analyzing system performance:
Ensure the system you are using is designed to support your workload. To determine
the minimum system requirements for your workload use the IBM Workload
Estimator.
To compare systems for Domino workload the system Mail and Calendar Users
(MCU) rating must be used. CPW is for traditional workloads and should not be used
for Domino. To determine the MCU rating for your current system refer to Appendix D
of the Performance Capabilities manuals.
Version 5, Release 4
Version 6, Release 1
IBM i 7.1
Response time is the best indicator of system performance. The guidelines presented in
this section are general guidelines limits where most customers start reporting a
response time degradation in their environment. Since each system is unique the exact
threshold where response time will be affected may be higher or lower depending on your
hardware and system usage.
189
Monitoring Tools

There are multiple tools available for collecting system performance.
Collection services
IBM i Job Watcher
IBM i Disk Watcher
IBM Systems Director Navigator
A discussion on the details of these tools and how to use them is beyond the scope of
this article. For an introduction on gathering performance statistics refer to Chapter 4 of
the IBM Redbooks Domino 6 for iSeries Best Practices Guide.

CPU
Keep CPU utilization at 85% or below for a single processor system
Keep CPU utilization at 90% or below for a multi-processor system
Be aware that a system can be CPU bound without showing a CPU utilization of
100%. How can this be? Imagine a system with 4 processors. If one processor is
pegged, but other processors are idle, the overall system CPU utilization will be 25%.
It is possible to have a single threaded, CPU intensive operation be running slowly
and thus CPU bound without an obvious indication that CPU is the issue. You should
review the CPU utilization for a single thread. If one thread is driving 100% of one
processor or 25% overall CPU in our example, then your system is CPU constrained.
Hint: Use the IBM Performance Tools for i5/OS WRKSYSACT command to see the
CPU utilization at the thread level.
By default all Domino jobs run with the same job priority, priority 20. There are cases
when you may want to alter the priority of some tasks. One example would be to try
and manage which jobs receive more CPU on a CPU constrained system. Changing
the run priority can be done by following the instructions in the technote Changing the
Priority of a Server Job Permanently on iSeries (IBM i). By changing the job priority
you can or force intensive processes such as view index updates, agents or adminp
to have lower priority over user interfacing tasks such as server and http.
Memory
Machine pool faulting rate and paging rate should be at or below 5 faults and 10
pages per second.
The memory pool for your Domino servers (the BASE pool by default) should stay at
or below a Non-DB fault/pages rate of 100/300 per processor. So for 5 dpars running
on a single logical partition or system with two full active processors should have
enough memory to keep the number of non-db faults/pages per second of 200/600.
Determining the correct amount of memory for a system can be challenging. One
simple rule of thumb is to check your largest database and most complex views. You
should have a minimum of 2 times the amount of space needed for the views in that
190
application dedicated for the Domino server. For example, if you have an application
database with multiple views that total 2GB in size you should have a minimum of
4GB dedicated to the Domino server in order for optimal memory performance during
view updates.
On a memory constrained system, until more memory can be purchased, reducing
the size of the NSF buffer pool can improve performance. Be aware that reducing the
NSF buffer pool may also reduce the number of router threads and you may need to
manually increase the number to prevent mail routing slow downs. For information on
how router allocates threads refer to technote 1093562.
Disk
When reviewing disk I/O and percent busy rates you want to see the disks to be 35%
or less busy.
Avoid running out of disk! Ideally your drives should not be 90% or more utilized.
The space used (%used) should be consistent across all drives.
When balancing drives the Domino servers should be ended.
The protection/mirroring status should be active, not degraded as a status of
degraded may indicate an under performing drive or possible disk and hardware
errors.
Domino Tasks

There are things specific to Domino that can help or affect overall performance.
Stay up to date on ODS level. It is not required that you update to ODS level 51 when
you upgrade to Domino 8.5. However; there have been a number of performance
improvements that will affect server performance and thus it is important to upgrade
as soon as possible.
Disable any unnecessary tasks. Are you running SMTP on your administration or hub
servers? Are you running COLSRV400 on multiple dpars? Are you running DECS
and not using it? Each and every task running under the Domino server takes some
system resources to use so eliminating unnecessary tasks will improve overall
system performance.
Transaction logging has evolved to be a valuable resource for all customers.
However; there is a performance impact to enabling transaction logs. In most
implementations a CPU increase of 3% is seen, for example CPU used by the
Domino server is 38% before transaction logging will be approximately 41% after.
There is also typically an increase in the I/O rates between 5-15%. Thus if the
percent busy rate of your disks is normally 10% busy you may see your drive %busy
rate at 10.5 - 11.5% busy after transaction logging is enabled.
For other overall performance information refer to Sizing Large-Scale Domino Workloads
on iSeries.
191
4.9. Tuning Tips for the Domino HTTP Server
The default settings are a "one size fits all" approach. While the settings will work as
shipped, making some minor changes based on how you use the Domino server can
increase the performance and efficiency of your server. Here is a list of things you can do
to optimize your server.
4.9.1. HTTP Server Threads
In order for you server to function properly it must be able to handle all of the requests it
receives. Domino provides a number of statistics to help you determine the health of the
Domino web server. For example, to determine the peak number of threads used by the
web server review the statisticDomino.Threads.Active.Peak. By default, the number of
active threads is 40. However; if you are using all available threads and have available
memory for additional threads you can increase this value in the server document,
Internet Protocols... HTTP tab, Number of active threads parameter as shown in figure
1.

The Domino web server queues requests as they come in so they can be processed by
the HTTP server threads. In some cases changing the queue method used can improve
performance. For more information refer to technote 1201715: HTTP thread queue
implementation in 6.x can cause performance issues for some setups.

192
4.9.2. HTTP Requests
When tuning your server, it helps to understand what requests are being processed on
your servers. You can do this by logging Domino web server requests. Thus, you can
access this information "after the fact" by reviewing the access logs and/or the web log
database domlog.nsf. You enable logging for your server in Internet Protocols tab of the
server document as shown in figure 2.

In order to see which requests are being processed in "real-time" you can use the
Domino command tell http show thread state. This will present you a list of all active
threads and the URL currently being processed if the thread is not idle. Below are two
example entries and you can see that the first thread is idle and the second thread is
opening the Inbox for User One (uone.nsf).

11/18/2010 12:09:28 Http Worker Thread ID [1c7]: [Thread State is Idle]
11/18/2010 11:18:12 Http Worker Thread ID [2425]: Working session [9a732]: Session
State [Processing Request] : GET
/mail/uone.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries&PresetFields=
DBQuotaInfo;1,FolderName;($Inbox),hc;%24Sender1%7C%2498,UnreadOnly;1&TZType
=UTC&Start=1&Count=23&resortdescending=1 HTTP/1.1

If your server is struggling to process all requests due to database contention, view index
rebuilds or database corruption, you will see all requests showing the same database,
view or document. This is a fast and simple way for you as an administrator to identify
potential database issues. In other cases, you may see that all of the pending requests
are agents. By default, only 1 agent can process at a time. However; if the agents
running on your server are thread-safe, then you may modify the configuration of the
server to allow multiple web agents to run concurrently. To make the change, open the
server document and access the Internet protocols... Domino Web Engine tab.
Change Run web agents and web services concurrently to Enabled as shown in figure 3.

4.9.3. JVM Heap
The default JVM Heap can be modified using the NOTES.INI parameter
HTTPJVMMaxHeapSize. If you are using XPages, you will want to increase the size of
the heap at 8.5.2 and higher. If you are not using XPages and running 8.5.1, you will
want to decrease the default size. More information can be found in technote 1377202:
What is the HTTPJVMMaxHeapSize notes.ini parameter in Domino 8.5 and what should
it be set to?

193
4.9.4. Database Performance
When working with the Domino web server and applications, the same techniques to
improve database performance from a Lotus Notes client can be used on the web. For a
list of the most common reasons for slow database performance refer to technote
1174563: Knowledge Collection/Troubleshooting Guide: Known causes of slow database
performance.

194

Redbooks Wiki Optimizing Lotus Domino Administration

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Redbooks Wiki Optimizing Lotus Domino Administration

Uploaded by

Copyright:

Available Formats

Redbooks Wiki: Optimizing Lotus

You might also like