Enterprise Vault Backups

SYMANTEC TECHNOLOGY NETWORK WHITE PAPER: RISK MANAGEMENT

Symantec Technology Network

Enterprise Vault Best Practice
Enterprise Vault Backups

- by Steve Blair Symantec Sr. Technical Product Manager, Enterprise Vault

December 21, 2007

Symantec Technology Network
1

1

Enterprise Vault Backups

Table of Contents
Table of Contents................................................................................................................................... 2 Overview ................................................................................................................................................... 3 Introduction ........................................................................................................................................ 3 Scope of Document........................................................................................................................... 3 Target Audience................................................................................................................................. 3 Acknowledgements .......................................................................................................................... 3 Backup Trends .................................................................................................................................... 4 Enterprise Vault Infrastructure and your environment ................................................ 4 Backups vs. Archiving ................................................................................................................. 4 Enterprise Vault Backup Requirement Overview ................................................................. 5 High-level Enterprise Vault Deployment .................................................................................................. 7 Enterprise Vault Backup Requirements .................................................................................................... 7 Enterprise Vault Server (1)........................................................................................................... 7 Enterprise Vault Indexes (3)........................................................................................................ 8 Enterprise Vault Shopping Service Locations (3) ................................................................ 8 Enterprise Vault Store Partitions (3) ........................................................................................ 8 SQL Databases (2) ............................................................................................................................ 9 Enterprise Vault Backup Timing and Scheduling.................................................................... 10 Overview of an evening for backups ........................................................................................... 11 Addendum / Additional Information ........................................................................................... 12 PREBACKUP.BAT:............................................................................................................................. 13 POSTBACKUP.BAT: .......................................................................................................................... 13 About the Author....................................................................................................................................... 15 About Symantec Technology Network (STN)......................................................................................... 15

Symantec Technology Network
2

2

Enterprise Vault Backups

Overview
Introduction
This best practice document discusses the inherent needs to securely backup operations Enterprise Vault servers to insure best available DR scenarios. With several very important information sources integrated with Enterprise Vault, it is critical that proper backups of this data be performed on a regular basis. This paper considers the impact this has on Enterprise Vault ways customers can easily implement a high degree of coverage for their archived content. It is important to understand that it is a Best Practice to regularly backup the components of your Enterprise Vault environment to insure that you have adequate DR protection as with any Tier-1 business critical application.

Scope of Document
This document is focused on Enterprise Vault, Microsoft SQL, Vault Store settings, and the indexes that are part of the environment as a whole. This document presumes a good working knowledge of your company backup products, schedules, and DR plans. The implementation, planning and use of actual backup software / hardware solution are outside the scope of this document. However, Best Practice recommendations to insure your DR plans protect your data in Enterprise Vault for use of your backup software are made.

Target Audience
This document is aimed at customers, consultants and support staff and it is assumed the reader has a good understanding about the architecture and operational aspects of an Enterprise Vault server, and their internal network and storage architecture. This document also discusses concepts related to Microsoft Server management, and expects the reader to either be skilled in this area, or have team member(s) who are.

Acknowledgements
I would like to acknowledge the contribution that other individuals made towards making this a successful and informative document. Contributions and feedback came from the following teams: Technical Product Management, Technical Field Enablement, Engineering, Consulting, and our Customer Support Teams.

Symantec Technology Network
3

3

Enterprise Vault Backups

Backup Trends
Companies implementing Enterprise Vault typically purchase to solve one of several needs. They are either trying to offset the explosive growth of individual mailboxes, file server personal / shared areas, or insure relevant regulatory compliance in their country. Backing up your data to insure your company has an adequate safety net has long been the bane of the modern System Administrator, but lack of backups place a company at extreme risk in a DR (Disaster Recovery) situation. Because of implementing Enterprise Vault, a company must undertake a review of their existing backup schedules and plans, as archiving allows customers more flexibility and newer ways to have a safety copy. At some companies they are of a smaller size; thus their Enterprise Vault backup scenarios are relatively straightforward to deliver. At larger companies, given a number of Enterprise Vault servers their backup scenarios are slightly more complicated. Given the advances in both large storage tape technologies such as DLT^2 and LTO3, and disk systems that operate like tape or replicate, no two customers studied are performing their backups in a similar fashion. The one unifying similarity of Enterprise Vault customers regardless of size is that they are aware they have to insure for their internal &/or external customers a high degree of data availability. Enterprise Vault Infrastructure and your environment As the landscape changes with the introduction of Enterprise Vault into your environment, it does not absolve one of the responsibilities to insure that adequate backups are in place. Given the data contained in Enterprise Vault is still just as valuable, just as important, it is our recommendation just like with email and file servers, that you treat your Enterprise Vault server(s) as any other Tier-1 application server in your network. Because of installing Enterprise Vault at your site, you will observe (in many cases) greatly shortened backup times for the server(s) that Enterprise Vault archives from, but you will still need to make sure your DR position is strong by the scheduled backups of your Enterprise Vault data. Backups vs. Archiving Let’s begin by reviewing that many people are confused over the two terms “backup” vs. “archiving” and often wonder if there is any discernable difference. After all, is a backup not an archive in time, or a backup stored offsite considered an archive? It is absolutely true that both terms are valid, but the line becomes clearer when it is time to produce content back from a backup. Backup media is not the same as a hard disk spinning online, and as such can and does suffer from media failure, inability to read / restore content, and data loss of the physical

Symantec Technology Network
4

4

Enterprise Vault Backups

media. The important factor is time to produce / restore and this is where archiving is unequalled in terms of performance and availability vs. a backup / restore solution. Active archiving differs from regular backups as it has indexed and represented the content in a way immediately available for recall, review and use without operator interaction. The very activity of having the content still stored, managed, and available replaces many daily System Administrator tasks, and supplements end-user experience of their data being highly-available via “active archiving” as delivered with Enterprise Vault. There are two major Best Practices you have available to perform a backup of your important information stored in Enterprise Vault: data-only backups of the archived content, and data and application backups of the whole Enterprise Vault environment. We shall review both in this paper, and allow you as the reader to take a decision on which will be the Best Practice for your environment.

Enterprise Vault Backup Requirement Overview
Let us begin by saying that there are several crucial components in your Enterprise Vault deployment that need to be backed up periodically. While having Enterprise Vault as your archiving solution greatly reduces your backup reliance, it does not absolve you of performing backups.

Symantec Technology Network
5

5

Enterprise Vault Backups

The items in your environment that must be backed up are: SQL Databases (Directory, Audit, Monitoring, Compliance &/or Discovery Accelerator, and Vault Stores) Indexes, their directories and files Vault Store partition (s) and files EV Server system for registry settings EV Server(s) license file (per EV server!) Any customisations done to your EV environment: Scripts Welcome Messages Webapp.ini PST Migration Templates Discovery Accelerator and Compliance Accelerator o Web-config files o XML templates o Custom Search word lists Each of the above components being backed up plays a vital role in the Enterprise Vault environment as a whole. It is a Best Practice to insure that good backups of the above environment are performed on a regular basis as part of your total DR plan. The diagram below shows a sample Enterprise Vault deployment and the interdependencies used by the various components in a simple deployment. Whilst this diagram may not accurately reflect the environment you have Enterprise Vault deployed in at your site, it does cover the important components of the Enterprise Vault environment. The theories and Best Practice guidance done in this paper are the results of years of testing, and actual customer experience to help guide you on making your environment very reliable. Part of the Best Practice of backing up your Enterprise Vault environment is to test the backups periodically to insure that your backups are good. Many customers have relied solely on others or receiving an email “the backups worked” and never test their backup’s validity. As with any Tier-1 application, this is a risk to your business that we strongly encourage you to avoid by performing regular simple checks of your backup solution. In many cases, customers adhering to these Best Practices have uncovered that there was a problem with their backup solution only after a hardware / server failure thus we strongly encourage you to chose a known industry partner’s products for backups of your Enterprise Vault environment.

Symantec Technology Network
6

6

Enterprise Vault Backups

High-level Enterprise Vault Deployment
Each item is detailed by number in the next section to fully explain the content and importance to your implementing Best Practices with backing up your Enterprise Vault Infrastructure.

Enterprise Vault Backup Requirements
Enterprise Vault Server (1)
Now that we have covered the main components of the archiving infrastructure in your environment, we have to consider your Enterprise Vault server itself. Because changes can and do get made (new archiving policy, new retention rules, new Roles-Based Administration controls), it is a Best Practice that the Enterprise Vault server itself receive a regular backup.

Symantec Technology Network
7

7

Enterprise Vault Backups

Content on your Enterprise Vault server that should be backed up is: System Registry Directory where Enterprise Vault is installed The entire registry for Administrator including local directories of: • Local_System • Current _User Any directory where you store any: • Custom Forms • EVPM scripts • Modified IIS configuration files While the Enterprise Vault server does not require nightly backups, given changes do happen in production, it is a Best Practice to perform a backup at least once-aweek on the above content.

Enterprise Vault Indexes (3)
Contained inside of your Enterprise Vault deployment is one to many AltaVista™ indexes. If you are unsure as to where your indexes are located, you can determine the location in the properties of your Indexing Service in the VAC (Vault Admin Console) Console. These indexes have one of the most important parts of your archived content and must be backed up as a Best Practice on a very regular basis. Scheduling of the backups is (covered later in this paper) very important to insure the system is at a quiescent state for optimal integrity of the indexes whist performing your backup. It is a mandatory Best Practice that you only backup the indexes when Enterprise Vault services are either stopped, or running in “Read Only” mode to insure an accurate backup of them.

Enterprise Vault Shopping Service Locations (3)
When users want to restore items manually contained in Enterprise Vault, their content goes back to a Shopping Service location. This location is recommended to receive a regular backup, but it is not necessarily required to back it up on the same nightly schedule. Whilst the Shopping Service is quiescent during your backup schedule, users will still be able to search content, but not restore it so it is a Best Practice that the scheduling of your backups occurs during user off-hours. Unlike other parts of your Enterprise Vault environment, it is not mandatory to backup your Shopping Service locations, but if you do perform backups of them it is a Best Practice that the Enterprise Vault servers are either stopped, or running in “Read Only” mode.

Enterprise Vault Store Partitions (3)
Your archived content stored from its original location into Enterprise Vault will typically be located as many flat files on an NTFS partition (exception: EMC

Symantec Technology Network
8

8

Enterprise Vault Backups

Centera) with a folder directory structure cascading down. If you are unsure as to where your Vault Store Partitions are located, you can determine the location in the properties of them in the VAC (VAULT ADMIN CONSOLE) Console. As the content in your Vault Store partitions is crucial to your environment, it is a Best Practice that all archived content receives regular backups as disks and disk subsystem solutions can fail. It is a mandatory Best Practice that you only backup the Vault Store partitions when Enterprise Vault services are either stopped or running in “Read Only” mode to insure an accurate backup of them. If the size of your Vault Store Partitions is very large, you may wish to consider options such as Symantec VVR, disk-replication (EMC Centera), SnapBackup (Network Appliance Filers) or RoboCopy to aid in reducing end-user downtime / access to archived content. Whilst the use of these technologies is outside the technical scope of this paper, they are all very good solutions that should be considered for larger sites as part of a Best Practice scenario for DR protection.

SQL Databases (2)
Contained in your Microsoft SQL environment are many databases that Enterprise Vault uses as part of the solution. These databases are essential to insure that your deployment is functional, able to archive and know where content is when you, or your users access archived content. Because of the number of users, some sites will have a dedicated SQL server for their Enterprise Vault environment; in other smaller sites, shared access to a SQL server. Regardless of SQL installation type, Best Practice dictates that you must make backups of the Enterprise Vault data contained in SQL. Of particular importance, the following Enterprise Vault databases as part of the Best Practices must be backed up on a regular basis: Directory Database Audit Database Monitoring Database Database transaction logs Compliance &/or Discovery Accelerator Database(s) For upgrades / Sp’s being applied to your SQL server, it is a Best Practice to make sure that when a full backup is made that the master and msdb databases are backed up as they contain information pointing to the existence of Enterprise Vault on the SQL server. In many sites, the above parts of your Enterprise Vault infrastructure may be backed up by a separate person, such as a DBA (Database Administrator) which can be sufficient but the scheduling (covered later in this paper) is critical for these components to be properly secured. When backing up SQL server(s), it is a Best Practice to:

Symantec Technology Network
9

9

Enterprise Vault Backups

Backup the Enterprise Vault databases at least once a week Truncate the transaction logs at the same time (to maintain reasonable disk space consumption for the overall SQL server health.) Backup the database transaction log daily Backup the Enterprise Vault databases before / after any changes to your Enterprise Vault environment (Service Pack, new release installation, new target(s) being added) Depending upon your Enterprise Vault architecture, you may have just one set of databases, or in the case of our Compliance Accelerator / Discovery Accelerator customers you may have additional databases. Both require that Best Practice controls as part of your DR solution that they are be backed up each day as well.

Enterprise Vault Backup Timing and Scheduling
The previous diagrams went into detail at the network level for the various components used within a simple Enterprise Vault environment. Timing is a crucial part of a Best Practice for backing up your archived content to insure optimal safety / DR preparedness for your environment. Many resources are contended with by the “backup window” in an IT organisation on a nightly basis, and each of these plays a role in determining how, when, and what to backup. Most customers are aware overall of activities on their infrastructure, but can fail to take into account other activities that impair their ability to regularly successfully get a good backup of their Enterprise Vault data. This can have side-effects during a DR scenario that no one would want to encounter. Important factors that can affect your backup timing include: Archiving Window o The nightly window when Enterprise Vault is converting content o When are users offline and online Network traffic o Is the LAN or WAN capable of backing up large data sets o What else is running over the LAN or WAN to the target server(s) to be backed up? How much time do you have in a backup window o Will any servers with Enterprise Vault data be unavailable o Contention with other network activities When does Exchange &/or Domino o Have its’ backups performed o Compress mail stores or databases Will the backup be performed to

Symantec Technology Network
10

10

Enterprise Vault Backups

o o o

Tape NTFS disk Centera replication

Each of the above factors can and do have a direct effect on your ability to perform your Enterprise Vault backups. Failing to counter in these type of network and system activities is against Best Practice and should be avoided wherever possible.

Overview of an evening for backups
In implementing your Best Practices for backups, we recommend you make a chart that details significant activities as discussed in the previous section. A suggested chart to give you good visibility is below.

Activity
Archiving Window MSMQ Buffer Drain Window Exchange Backups EV into Read Only Mode EV Archive Backup EV into Normal Mode EV Pending Shortcut Conversion EV SQL Database and Transaction Log Backups

Start Time

End Time

Best Practice Recommended Frequency
Daily Daily Daily or Weekly – site specific At Start of each time backups are performed As dictated by local site rules, at least once a week At End of each time backups are performed Can run from end of Backups to 0900 / user arrival time in mornings or as part of archiving task scheduling – site specific Daily – Transaction Logs, Weekly EV Databases

EV Server Backup

As dictated by local site rules, at least once a month

The above chart is very handy to assist you in making sure that you have proper justification, and inform all on your team of the activities to hand. The recommendations in the chart vary slightly if you have Enterprise Vault safety copies enabled, and would allow you to avoid the Pending Shortcut conversion for sites that have chosen that option in the VAC (VAULT ADMIN CONSOLE).

Symantec Technology Network
11

11

Enterprise Vault Backups

Addendum / Additional Information
It is important to know that several additional items in your Enterprise Vault environment can have an effect on your backup windows, and should be reviewed periodically as part of Best Practice to insure that you meet your SLA to your business. Tape & Partition Management for LTO3 users • If you setup your maximum partition size to be slightly less than the current size of one tape (400GB) it will make administration and management better for your time windows. Backup closed partitions • Open partitions should still be backed up daily or at most weekly. • Once a month if you have collections enabled as when an item in a collection is no longer retained, the collections can decrease in size. Refresh tapes • In sites where you do not do disk-based backups, we recommend you cycle new tapes into your Enterprise Vault backup schedule as dictated by local site rules. • Insure proper tape destruction procedures are in effect to protect IPR (Intellectual Property) contained on these tapes. Scheduling the start and stop of archiving • Allow your Enterprise Vault server and MSMQ queues sufficient time to finish any current activities so that you have an optimal backup image taken. • Factor in how much you can do in parallel. Use a good batch script to run backups • Insure that all Enterprise Vault tasks must are setup to run on the Enterprise Vault server as type “Automatic” • We recommend as a Best Practice the script below from our Enterprise Vault documentation at a minimum • Allow a minimum time of 10 – 15 minutes before / after each script before starting backups. The following scripts are provided as an example to guide in performing Best Practices to prepare your Enterprise Vault environment for backing up the data contained in it. The script “PREBACKUP.BAT” stops your Enterprise Vault services from archiving to be ready to backup content. The script “POSTBACKUP.BAT” changes Enterprise Vault back into normal operational mode once your backups are done.

Symantec Technology Network
12

12

Enterprise Vault Backups

PREBACKUP.BAT: REM --------------------------------REM prebackup.bat REM This script is to put EV into read-only mode so we can run backups REM --------------------------------net stop /y "Enterprise Vault Task Controller Service" net stop /y "Enterprise Vault Storage Service" net stop /y "Enterprise Vault Indexing Service" net stop /y "Enterprise Vault Shopping Service" regedit /s c:\readonly.reg net start "Enterprise Vault Storage Service" net start "Enterprise Vault Indexing Service" net start "Enterprise Vault Shopping Service" net start "Enterprise Vault Task Controller Service" POSTBACKUP.BAT: REM --------------------------------REM postbackup.bat REM This script is to put EV into read-write mode after running backups REM --------------------------------net stop /y "Enterprise Vault Storage Service" net stop /y "Enterprise Vault Indexing Service" net stop /y "Enterprise Vault Task Controller Service" regedit /s c:\normal.reg net start "Enterprise Vault Storage Service" net start "Enterprise Vault Indexing Service" net start "Enterprise Vault Task Controller Service" Customers using Enterprise Vault FSA (File System Archiving) need to run an additional script when a 3rd party backup application accesses a file-system with Placeholders on it so the application will respect the “Archive Bit” flag that FSA uses. FSABACKUP.BAT REM --------------------------------REM fsabackup.bat REM This script is to put FSA into non-recall mode to allow applications to REM backup a file system containing PlaceHolders REM --------------------------------EVFSABackupmode.exe backup (or normal) <Server> [DirectoryComputer] REM The above script must be run in “backup” mode to stop the Placeholder service, and “normal” mode to start the Placeholder service on each target FSA server.

Symantec Technology Network
13

13

Enterprise Vault Backups

Symantec Technology Network
14

14

Enterprise Vault Backups

About the Author Steve Blair was the founder of Product Management in KVS before acquisition by VERITAS in late 2004. His role has been pivotal in helping the Enterprise Vault team expand and grow the products over the past two years. Steve’s responsibilities include Product Strategy, Core functionality, Storage, Internationalization, and Product Evangelist. Presently Steve is the Sr. Regional Product Manager for Enterprise Vault in Asia Pacific region and is based in Sydney Australia. Prior to joining KVS, Steve was involved in technology assessment and advisement for a variety of UK and EMEA based Venture Capital firms advising investors and boards on product strategy, and how to best grow emerging technology firms from concept to reality. In prior positions, Steve was a Global Director in IT for a multi-national German bank for Enterprise System Management for the investment division of the bank. He occupied the role of Chief of Technology for a MoD certified Managed Services provider across EMEA. In other startups, Steve spent many years from pre to post IPO one of the Principal Engineers doing R&D for new products in areas of distributed systems theory, as well as network security and was lead or co-inventor for many patented inventions to aid IT groups in more efficiently doing their jobs. He has over 22 years of experience with IT and development activities. He has worked in Hardware and Software development, Systems Administration, Network Operations, and Enterprise Systems management. About Symantec Technology Network (STN) Symantec Technology Network (STN) is Symantec Corporation’s technical information generation and dissemination organization. It distributes a free monthly technical newsletter that discusses timely technology events to 120,000 email subscribers across the globe. STN also publishes technical data storage and security articles each month for large enterprise and Small and Medium Business (SMB) readers, as well as hosts a variety of blogs and product discussion forums discussing Symantec product tips and insights. To subscribe to STN’s Technical Newsletter and review other STN materials, please visit STN at: http://www.symantec.com/enterprise/stn/index.jsp W. David Schwaderer is STN’s Editor-In-Chief. He has authored ten technical books, and six commercial software programs. His eleventh book titled Innovation Survival is nearing completion. He has a Masters Degree in Applied Mathematics from the California Institute of Technology and an MBA from the University of Southern California. He lectures at Stanford and in Silicon Valley on the subject of innovation. Russ G. Gregg is STN’s Video Content Producer. He is presently focusing on producing stateof-the-art digital video for the Web and small form-factor, portable video devices such as Apple Inc.’s iPhone™ and Video iPod®. Russ has a Bachelor of Arts in English Literature, Texas from A&M University. Symantec Technology Network
15
15

Enterprise Vault Backups

About Symantec Symantec is the world leader in providing solutions to help individuals and enterprises assure the security, availability, and integrity of their information. Headquartered in Cupertino, Calif., Symantec has operations in more than 40 countries. More information is available at www.symantec.com.

Symantec Corporation World Headquarters 20330 Stevens Creek Boulevard Cupertino, CA 95014 USA 408 517 8000 800 721 3934

Symantec and the Symantec logo are U.S. registered trademarks of Symantec Corporation. Other brands and products are trademarks of their respective holder/s. Any technical information that is made available by Symantec Corporation is the copyrighted work of Symantec Corporation and is owned by Symantec Corporation. NO WARRANTY. The technical information is being delivered to you as-is and Symantec Corporation makes no warranty as to its accuracy or use. Any use of the technical documentation or the information contained herein is at the risk of the user Copyright © 2007 Symantec Corporation All rights

Symantec Technology Network
16

16

Sign up to vote on this title
UsefulNot useful