:I '="

Ir- •

W HIT EPA PER V E R I T AS""

BEST PRACTICE

BACKING UP AN ENTERPRISE VAULT ENVIRONMENT WITH NETBACKUP

VERSION 2.9 1/17/2005

'r:yh,OI''';'.'' H':~';:;;' r':::':"'ii":",;:)_i;':; '~'()[r:;_;£;r;;"::;~s :S0sA,,,,,t,,,,,,· <3(l"(j q! ;;,;'.1; iet .~;f·;;,~·n-~:~: :~~: it.'J,:,,~·,t (.::; ,~;:ht~nc;c ;,:.':U·:(',;r nci.<.;"

1

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044359

" "":'"

._ ,.

VE RITAS™

TABLE OF CONTENTS

Executive Summary 3

Defining an Enterprise Vault Environment 3

Systems and Services that need to be protected .4

Exchange Servers 4

SQL Server 2000 5

Share Point Portal Server and Windows SharePoint Services , 5

File and Print Servers 5

Enterprise Vault Server 6

Enterprise Vault Server Services , 7

File System Agents : 8

Vault Stores 9

Index File Locations , , 10

Shopping Service Files 10

Journal Service 10

Compliance Accelerator 11

Discovery Accelerator II"

Special Considerations for JoumalMailboxes .............................................................•.................................. 11

Backups ; 12

Traditional Backup Methods 13

Advanced Backup Methods ; 13

Backup's without Snapshots..................................................................................................................... 14

Backups with Windows 2003 Volume Shadow Copy (VSS): 16

Backups using NetBackup Advanced Client, Snapshot techniques and VERITAS Storage Foundation

for Windows 20

Recovering Enterprise Vault 21

Conclusion , 22

Appendix A 23

Example batch files for backups of an Enterprise Vault site 23

Backup_ev.bat. ; 23

Backup evvss.bat 25

Using the Windows Scheduler 28

Appendix B 29

AppendC 30

Installpath\netbackup\DbExt\MsSql\EVBkup.bch ; 30

Installpath\netbackup\DbExt\MsSql\EVincrbkup.bch SQL Agent Script 31

1(~>: .. : v·:·-(? \/cr::!T/>.'S. VCH~T:\S ~;:/:~::i":-': ;:',>:: Vf:~:~_[TAS ~(;)(l ':::"'<1 .:::,J O~i"(": VEF.iT;:\S j!',s.:.; us .. x:j':,~:! (ri-;'~;r (;-:>i;r::':(~ ,C1i :: ;;i::';):': l r:::)"''>:'-b ~., <F~): ~~l:):.\·::r:b ;,;GI':.I:Dn(-;·j hu({:=,(,

;)'1c:{,!at<~: ~'H'o:' i ;-an·::rn'::·F!<~' :.>r U, ; ,·~v:~tf:,:~~~.1

2

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044360

4 ";'"

• T

VERITASTM

Executive Summary

This white paper is intended to cover the nuances that Enterprise Vault presents to a VERIT AS N etBackup environment. Basic backups of the operating system and Windows Active Directory are assumed to already be in place as per regular backup practices.

Examples from simple traditional backups where all applications are shutdown to a quiescent state and then everything is backed up to much more sophisticated techniques that include hot backups, with snapshots and offhost backups.

This isn't intended to replace existing VERITAS documentation but to augment it by offering different ideas and making the backup architect aware of how best to implement a backup solution for an Enterprise Vault installation.

Defining an Enterprise Vault Environment

In order to properly implement a backup plan for an environment, the first step is to understand what needs to be protected.

An Enterprise Vault environment consists of potentially multiple systems, some of which maybe geographically disbursed.

A typical Enterprise Vault environment consists of a SQL 2000 Server, Exchange Servers, Enterprise Vault Servers, file and print servers, and SharePoint Portal Servers. For Exchange users there are also client

Users

C).r

~

I

I

Exchange

SharePoint Portal

File/Print Server

Sal Server 2000

Enterprise Vault Server

·<:;·p::.:r'f'( .::D:~:::) ',;LF~iT/<-: ':::;Oit;::;.'dXf; <::~·:)<::'·ai'k-;n ,..\H

:'.;;q;,~(::;:;.j n·'::<·;"::';:H"k~~ '/ VFr~jT/<:; (::Qfk .. ~:::u tn j'i":b U~;

GEORGE W. BUSH PRESIDENTIAL RECORD

3

OAP00044361

. ":"

. ~

VERITAS'"

extensions for Outlook, and/or extensions for Outlook Web Access. Customers using Microsoft Exchange Journaling typically have a separate Enterprise Vault server dedicated to servicing the Microsoft Exchange Journal server.

An Enterprise Vault installation contains multiple databases; the Directory Service database contains the . configuration information for every component and service within a specific Enterprise Vault site.

Additionally, each Vault Store will have a separate SQL database, this allows for scalability of the Enterprise Vault environment.

Systems and Services that need to be protected

Exchange Servers

• Exchange Information Store

o Use the NetBackup Exchange Agent

o See the appropriate NetBackup manuals for setup and configuration for the VERITAS Exchange Agent backup software.

Use the Exchange Agents for "hot" online backups of the Exchange Information Stores. Both full and Mailbox backups continue to be supported. Enterprise Vault uses the Microsoft Message Application Programming Interface (MAP I) to pull data from the Information Stores so nothing is loaded on the Exchange server. (Enterprise Vault can also return data back into the Information Store.) Using NetBackup with the NetBackup Exchange Agent will be no different from the typical Exchange environment without Enterprise Vault, except that the Exchange server will have less data to be backed up.

Enterprise Vault helps reduce the backup burden in four ways in a Microsoft Exchange environment.

1. Archives messages and attachments out of Mailboxes reducing the size of the Information Store using age-based or quota rules. A defrag of the Information Store is needed to reduce the actual size of the Information Store, Enterprise Vault removes data which increases the "white space" within the Information Store". New messages/attachments will use the "white space" area first.

2. Archives older messages and attachments out of Public Folders which reduces the Public Folder size using age-based or quota rules.

3. Archives Microsoft Exchange Journal messages and attachments, reducing the size of the Information Store. In addition archived content can be stored on Write-Once-Read-Many (WORM) devices to help meet corporate governance requirements.

4. Ingests PST files into the archives enabling compression and single-instance storage for most items.

Reduces the backup and restore burden of backing up multiple constantly changing PST files. By ingesting PST files, users and corporations can discover and search the content of the archived data.

<>PV1\fh .;J);>:~ '.jFT:Cf':<:. SC!t':'_':'ii'{-; '-:>::.,p·":';":-:d!(,,': ,t.y ;:)hi~-.; :;:-?S{-:,:·<~ _':/f: :.:~ T»; VL;\: i s-. >;(-: \;r' UT./<; fc}c n;':(~ dB C!>,:-f VE:n.n ;~,S )!G·:ju·::.i ::1:._:-r,G~3 ~~nci ::;.i0qGX;~; Hr~~ =f':,w·;.n',t·'i·V::, ~:A

'k(i-;?;;,;:;,rk.;, o! "<JFPiT!<:; H;th",-!.<~·J CUl (.'i.;(t,t;;;r, 'n 1->': l.iS y:jj::,: G\i, ; "-;<.;n!" ':, ( ~h; ~>>(l=.'U :':::':~'.;:., :n::Jiy ~~;~)~:i~·;r:L ;(;f.;r:iiG(l~·;·j h£·n:;lL ([:~.;y iY:; 1X:::d6::"Yi<;':. or ;-egistl-?,'H,j

4

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044362

c ~

, .

VE RITASTM

SQL Server 2000

• SQL Server 2000 database

o Shutdown SQL and do cold backups

o Use the NBUIBE SQL Agent

There are two types of databases used within Enterprise Vault. The first database is the Directory Service database. This database is the center of the configuration since it holds all the configuration information. This includes which services are running on which servers. The second type of database is the Vault Store database. For scalability and resilience each Vault Store has its own SQL database for all the metadata pertaining to the objects managed within that specific Vault Store. As new Vault Stores are created make sure that the backups are configured to backup the new database. For each managed object the database record is approximately 250 bytes.

Enterprise Vault best practices recommend that SQL backups be done daily after the scheduled running of the Archiving Services. Backu s of the S L tran action logs should be done more frequently (multiple

times per da. c up e Directory Service database at least wee ndits transaction log at least daily.

Additional bac ups e ore and afte configuratIOn c anges an upgra es are highly recommended.

p~J;t-!l__~ '.{: ~~~~~"'f-' Share Point Portal Server ariCfWindows SharePoint Services .

• SharePoint Portal Server

o For SPS 2001 refer to the VERIT AS Netlsackupt'" 5.1 for Microsoft SharePoint Portal Server 2001 System Administrator's Guide for Windows

o SPS 2001 is currently supported by Enterprise Vault with v5.0

o For WSS 2003 refer to the "SHAREPOINT 2003 SUPPORT USING NETBACKUP FOR MICROSOFT SQL SERVERTM" at

http://ftp.support.veri tas.com/pub/support/products/N etBackup DataCenter/264854.pdf

o WSS 2003 will be supported in v6.0 of Enterprise Vault

File and Print Servers

• File and Print Servers

o Traditional backups

o Synthetic Backups

o Snap shot technologies

Note: NetBackup Flashbackup shouldn't be used on Enterprise Vault File System Agent managed volumes.

·<>:::'pyriqhf .::O(i!) VEFUAS ~)cjtw~;;n:~ ~.:(··~j)()"~:~kn .:'\~; H'»;:'\_:":C~ \jL;:,:lT'::'.~;. \iEF~\'; ~.:~) ~;'/tif':~·:~·'~·' n',(: VEH~TA::3 <:';":(~ {·'d cl!',;~·:rVERjTl,:~; ~;rG·:h:·)

:;·!.<,-:>;(:~·>t .. , ,:,f VEYdT/\S ~)'.::fL:::!:::{J in ~>'.; U~; .:2«/::>; (:~~:':;{ (-Ci:ln?::'<~ C=~~'.=':,r i» c<L:U r:';('::,':<) ,:!o:~{Jr,r, ;(;f;l~EG(!~;,j f"£!(f;:(! m::·,y ))~'!

ct i.t';f::;' "i]:<):?r;i,·x; (_'-:::"::~·;£\:·:l.i)'~ ,:' .~~~ <: ,.·]·:;""'t t.. ';"\'::'"\/:" :,<:."n n)t

5

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044363

. ":"

. .

VERITASTM

Enterprise Vault Server

The Enterprise Vault Server is wherethe bulk of the processing occurs. Larger sites may have multiple Enterprise Vault Servers running. For example sites with high levels of email traffic that need to have Exchange Journaling should have a dedicated Enterprise Vault Server to service the journal mailboxes. The Enterprise Vault Journaling service by default runs every 60 seconds looking for messages and attachments to archive. In order to keep the journal size to a minimum, the archived data needs to be safe guarded by either using redundant secondary storage devices such as two Network Appliance NearStore's or two EMC Centera's, or making multiple backup copies per day, to clear the archive bit, so the journal mailboxes data

[0' can be freed. Adv.anced backup techniques like snapshots can greatly reduce the amount of time the '\ Enterprise Vault services need to be quiescent for backups. ~

As data is managed from Exchange Information Stores, File System Agents, SharePoint Portal Servers, or

from a PST ingestion process, all data is moved to the Enterprise Vault server(s). Multiple Enterprise Vault servers can be created within a Vault site for scalability. The main function of an Enterprise Vault server is to host services and rovide stora e locations called Vault'Stores, Inde~s, and Snopping folders. A"Single Enterprise ault server can have multip e au t Stores. Each Vault Store is compnsed of one QL database, an Index, and at least one Vault Store Partition. (The SQL database should be for performance ~reasons on a separate server from the Enterprise Vault server.) A Vault Store Partition is a physical storage

location. Only one Vault Store Partition can be open for writes (new files) at one time per Vault Store. For

sCalability multiple Enterprise Vault servers and or Vault Store Partitions can be created. The Index location can either be shared where multiple Vault Stores can use the same Indexing location or a

separate Index can be created per Vault Store. Vault Store

he Indexing directory,.is where the Alta Visa

~ dexing engme creates the index for each

anaged object. Typically one Shopping folder IS created per Vault server and shared among multiple Vault Stores.

Typical configurations will have multiple Vault Stores. Enterprise Vault best practice recommends separating the Journaling Vault Stores and Indexes from other Vault Stores and Indexes due to the volume of data.

Enterprise Vault best practices suggest that Vault Store backups be done after the nightly Archiving Service

run has completed. For traditional backups (those not using snapshots, and hot bac iques) it is

required that all the Enterprise Vault services be stopped during the backup. S e Appe~for how to stop the Enterprise Vault services.

Enterprise Vault by default, copies data to a Vault Store, and waits for the archive bit to be cleared from the copied data before it frees data blocks within the Information Store, SharePoint Services document libraries

.:~L'«~ VFF;fL« ~::.cjh<:::/f; ;(<j::C'i"(:;:-k<.:' ::t\/CF: IT/<::

6

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044364

sure that not only isthe Vault Store Partition up but that the SQL ual.au.:>.., ...

folders are also backed up at the same time to avoid synchronization issues.

The Enterprise Vault server runs multiple services which communicate via Microsoft Message Queues with Agents and also Core services necessary for managing data. Stopping a single Archiving service for one server doesn't affect other Archiving services. Users can also continue to read from a Vault Store as long as the storage service hasn't been stopped.

------- - -- . .--.---.~

These services are broken down into two classes of services, agent services and core services. Agent services act as couriers of data between Exchange, SharePoint Portal Server, File Systems, and the Enterprise Vault server. All agent services contain the name of the machine they are servicing. Core services provide the handling and management of data within the Enterprise Vault server.

Each Enterprise Vault Server in a vault site has at least some of the following services running.

Core Services: I

Directory Service Storage Service .j" Indexing Service 0 Shopping Service ~ Admin Service ff

Agent Services: /

Archiving Service V Retrieval Service Journaling Service Public Folder Service SPS Service

SPS Pruning Service

File Placeholder Service Discover Accelerator Service Compliance Accelerator Service

r~ault Directory Service enables administration of an Enterprise Vault installation by managing ! configuration information. This service utilizes the Enterprise VaultDirectory Database.

I

Storage Service manages the storage of items archived by Enterprise Vault.

. Indexing Service indexes items stored by Enterprise Vault which enables users to search for archived I

\ ... items.

lc)G -:.~:'::::i .::·d Gii~t··: \/fJdTl~5 ~)r(;:ju;~i :~<~mfn :":tn:i ::)Dqr·)_n~; ~':i'~~ jr,:w··;rn.;:·U-:~, (" ~;l(:(iU( ';'lG{);:')nr~ :(,~·)nt:Y:.G:j riUf~:;:(l r:~:..;y i.).:I;,: tr;:·;;.;if;::;drH-; (j( ·::>q~-:~l>:_:n:;(i

7

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044365

p ~

VERITASTM

Shopping Service enables web users to build lists of items that have been archived by Enterprise Vault. It uses a Retrieval Service to restore items to a Microsoft Exchange Server.

Admin Service provides control and management of Enterprise' Vault services and processes.

~\ .. Archiving Service scans a specific Microsoft Exchange Server for items that are ready for archiving and passes tho.s~ items t~ other s~rvices for indexi~g and storage. Each managed Exchange server will have its

. own Archiving Service runnmg on the Enterpnse Vault server.

~. .

Retrieval Service enables items stored by Enterprise Vault to be restored to a specific Microsoft Exchange Server. Each managed Exchange server will have its own Retrieval Service running on the Enterprise Vault server.

Journaling Service is used for journal mailboxes, one per Exchange server.

Public Folder Service services one Exchange server's public folders.

SPS Service enables items to be archived from a SharePoint Portal Server.

SPS Pruning Service is usedto manage out older versions of data from the SPS.

Discovery Accelerator Service is used for litigation investigations of archived data.

Compliance Accelerator Service is used for sampling of archived data from Exchange servers.

File System Agents

The Enterprise Vault File System archiving service works differently than the rest of the services like the Archiving and Journalingservices.

7il Vault· [Console Root, fnterpnse Vault, Dltectary on 5R2(', sanrack2 '(omputers'.,sr2<.sanrack2.com (SR2[)] I!I!l E3

Cons Ie t I' - '.' ... .. " .,.., i'oe5C1ition "~rus

:"J t ~e~:e Vault ~~~a:::"e"";rpffie"";' ....;va....;u~....;sh-~-' .... g Se-rv-ice"'O"""""""'--"Ente=rp~ris~e V-au-'t S-hDppJ....;·-rg-Ser-vic-e ---1:"::Run=nmg:!.' -..l.-"""'-"""'-"""'-__._"""'-"""'--------------'---I

H ~ Directory on SR2( ~Enterprise Va~ Indexing Service Enterprise Vault Indexing Service Running

8· i sanrack2 ~Enterprise Yau~ Retrieval Service for SR2A Enterprise Vau~ Retrieval Service for SR2A Running

lti ~ Archives ~Enterprise Vau~ loornaling Service for SR2A Enterprise Va~t lOlJrnaling Service for SR2A Running

:: '-£l Vault Stores . ~Enterprise Vault Archiving se"ice for 5R2A Enterprise Vaut Illchiiing Service for SR2A Running

iC' '0il' Co"lJUlers Ii&.

o '- •• ~ sr2c,sanrack2,cc <;IEnterprise Vau~ Storage Service Enterprise Vault Storage Service Running

8 f:i:I File Servers

. f) '9 \\Sr2b

'tG$

e{fJ File Archiving Policies 1.. {fJ Volume Polities

, {fJ Folder Pohcies G~ Retention Categorie!

I !.l

:'~(i'~~~·,;.n':;,i t~·i:<1·-:>l:,(:l:k'~ ':f \/LF\T.~\~:; :::(;f\';',·L:(·"; !;'··;~~<:!r:·i<·~ ; ';" ::i1 ~h··';: :i}>·!·'(~r'i:,;,;:' ':'_~:"'?i_k~n;!;,.:;'

8

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044366

9 ~

- ~

VE RITASTM

When a File System is configured for management it doesn't show up in the same area as the Archiving or Journaling Services. The File System Agents show up in the File Server section of the GUI. There isn't a specific Archiving Service running to serve each managed file system server like the Enterprise Vault Archiving Service for Server SR2A shown above.

Instead they are listed under the File Server section of the GUI. Each configured server like SR2B below will be shown along with the specific volume that is being managed. The Storage Service works with the File Servers to archive data. For synchronization proposes if the Enterprise Vault environment only has File System Agents, for consistent backups the Storage, Admin, Indexing, and Shopping services need to be

stopped before snapshots or file level backups are started. .

'i'j) Vault· [Console Root'<Enterpnse Vault\Dlrectory on 5R2(' sanr8ck2 "flle 5ervers\\ \5r2bJ 1!I1!l13

[jCoosoieRoot

i3 • Enterprise vd 2' Dife<:toryon SII1C

'2 t sarrw2 [E·rte Archives

it; D vatt Stotes [3f!j Computers

, : ~ sr2i:.mad<2.cc H m fie Servers

. Plk,·ft

: ,- 1;11

'.. G$

i?,'FieAt1:hivi1gPolicies , volurne Pohties

'. , Folder Policies C0 Retention Categarie!

Note: When backing up the Vault Store partitions, Indexes, SQL database, and Shoppingfolder directories for a File System Agent the Admin service also needs to be stopped for quiescent backups.

All other Vault Stores are configured on the Enterprise Vault server as either local drives (NTFS), SAN attached (NTFS), or like NetApp NearStor Snap Lock volumes, or UNC Shares.

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044367

· - - "

VE RITAS™

and the Microsoft Archive file attribute has been cleared. (Both NBU and BE clear the Archive file attribute after backing up a file.) This safety feature can be turned off.

Note: For testing purposes simply executing the "attrib -a /s" from the command line will clear allarchive flags from all files and subdirectories.

\:1..<> format of a Vault Store is the name of the Enterprise Vault Store, the Vault Store Partition and then the l_4; time directory structure. An example is: V:\Enterprise Vault Stores\VaultPtnl\2004\11\15\18

This equates to volume V, Enterprise Vault Stores directory and VaultPtnl is the Vault Partition number. The 2004 is the year, 11 is the month, 15 the day and 18 is the hour expressed in GMT, when data was

~ .._

~ ""

.Y r Note: The directory s.tructure doesn't allow simply doing a backup of the current day,.since it's the creation

~ time of the data not the archive time of the data that is used for generating the directory structure.

Note: A single Vault Store can have multiple physical partitions but only one will be openfor writes at one time. For example, a single Vault Store may have 4 lOOGigabyte volumes, G, H, Land J As the archive grows Vault Store Prtl (drive G:) fills up, Enterprise Vault stops writing to Prtl and opens Prt2 the H: drive for new archives. This presents a few things to be aware of when setting up a backup plan. Make

sure all the Vault Store Partitions are backed up, and that some a the artitions WQ 't be etti

data but maybe having data deleted from t em, as it ex ires or is deleted. The NetBackup True Image

Recovery op ton cou e use on au are Partitions, In exes and Shoppingfolders to help

prevent too much data from being restored!

Index File Locations

Each Indexing Service can store its indexes in multiple locations. Use the Enterprise Vault Administrator Console to examine the properties of each Indexin Service to determine which folders need to be backed up. The default location is rogram Files>\Enterprise Vault\Indexing

Shopping Service Files

Each Shopping Service stores shopping basket information. Use the Enterprise Vault Administrator Console to examine the properties of each Shopping Service to determine which folders need to be backed

up. The default location is <~terprlse-~a:~~~~

Journal Service ---- .. -.-----

The Journal feature of Microsoft Exchange provides a full journaling of all email into and out of a specified email box, to a special designated journal mailbox. The journaling is done in parallel with the user sending and receiving their email. This is typically done for regulatory compliance such as for Sarbanes-Oxley .

'<':'l):/r'f:~ .:~G:Y) VLF'rr!~:.:, ~;.cit'.;:·,,~·:trG C;_""~)(;'<'l<;i(::~ :~;(::~:r··:-(;:;j !~i.<··:>rn~·l)''.~ ::>i VLHiT/~3 ~<;iL~·(.~'G

,-~·>,(-, .... :··?u VU:·:1Ti\S )f::HiT'<~5 ~~,::h';·f·'i'·':· ,:", ~;~(; U~; d.ndi::,; Ui>:-' <-U\.;i:~::';;r~ C;il~~;'i ;;l'";(L.::;l

.;:-n(.! ~·,H (;H~:;:;,: VEH~T/\S prc;.jd·) nar(:GS :·y,d ~·~rt0 :L:w,·:~mHrv.s ,:'r

':)~iX';(:~'" ;(~(·;,';tiD(:H·j hu(~;;:~~ (U;:y !Y:; ~(:::de:'!~)(;" h : ';~~r"3l~~:G-::i

10

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044368

, ~

VERITAS™

Compliance Accelerator

The Compliance Accelerator enables organizations to monitor employees' messages to ensure compliance or good business practice. This is typically used at brokerage houses to monitor email for various rules and regulations. It provides two main ways of monitoring email: Random sample of each employee's messages can be captured and sent for review each day. Or all messages can be searched for words or phrases that may indicate unprofessional behavior. The Compliance Accelerator adds an additional SQL database to the Enterprise Vault site that needs to be protected. The Compliance Accelerator Service should be stopped to provide a quiescent backup. Consult the Compliance Accelerator Installation and Administration guides for specifics.

Discovery Accelerator

Discovery Accelerator enables companies to conduct searches of archived mail and documents in response to an legal discovery. Discovery Accelerator enables the companies' legal team to review items found by the searches to determine their relevance to the case. Items are marked as being relevant to the case can be exported to be used as required. The Discovery Accelerator adds an additional SQL database to the Enterprise Vault site that needs to be protected. The Discovery Accelerator Service should be stopped to provide a quiescent backup. Consult the Discovery Accelerator Installation and Administration guides for specifics.

Special Considerations for Journal Mailboxes

As litigation continues to increase more companies are moving toward journaling all mailboxes, which increase the strain on the email infrastructure'. With the EnterpriseVault Journaling Service all the messages and attachments are quickly moved out of the Journal Mailbox into the Enterprise Vault Store. Depending on the reasons and regulations the customer is addressing by using journaling the Vault Store partition devices may need to be WORM (Write-Once-Read-Many) compliant. Enterprise Vault supports several WORM compliant devices, such as Network Appliance NearStor with SnapLock, EMC Centera, and Pegasus WORM Optical and WORM UDO. These devices at least for specific periods oftime can't be erased or rewritten until the storage retention time is over.

The Journaling service by default, starts an archiving pass every 60 seconds. Every message and attachment is archived to the Vault Store. This puts a large load on the Journal Service, Index Service and Vault Store, which is why it's suggested that aseparate Enterprise Vault server be used just for Journal servers. There are several reasons why best practices suggest that Joumal Services be done on a separate Enterprise Vault server.

1. Volume of messages and attachments managed per hour.

a. Includes single instance storage processing

b. Indexing every message and attachment

c. Updating SQL entries

d. Expiration of data if applicable, many customers never expire any data

i';("1)'-./f")h .TC"; \l;::;:~iT!"~S ~>cft'";v~:~r(-) : l.Y!' ;:~~.~~~ {";'1 'H:'( i·:.>; ((~vk':, ··'f '"/LF l'F /~:~ Set: ';\:f.~r (; ~: ··;::1;:~r:":;v·1.~~ iA ~~"~ ;1" :'(:~':P::"i~!:

,,:tJ~:T/:S k:y'i ,::.::',(j :.~lH cj1~u VCJ<.nr.S prn·lu·:j n .. ;;;"";(>1 ~:::n;j '-)jG~r::n;;.: ;"F':~ ;.f"·:~(j~ff(;~:,d<~,"

'H,("(;:{ <:~,:~~.;(:() ;n':";(:~.~GnG·j h~"J(~:;l(; ,F:l"r 1;::·' t(:'·i;k·;'::~Y}:~~ ;'?~WA>:".:"")<l

11

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044369

· -

- ...

VERI

2. Backup requirements are different than the regular Archiving Service w once per day

a. Journal mailbox data isn't freed until the archive bit has been cleared or a secona copy of the data has been detected in the case of two Centera devices.

i. MUltiple backups per day maybe required to keep the Information Store to a reasonable size.

Note: When using a separate server for Journal mailboxes make sure to also configure the Storage, Indexing, and Shopping service also on that server!

Backups

There are many of ways VERIT AS backup products could be configured to help protect an Enterprise Vault environment. Below are just a few examples, along with information about what needs to be safeguarded. For specific information about implementing a backup strategy for an Enterprise Vault site, consult the VERIT AS Professional Services&:nterptj se~ult Administration Guide, and the Enterprise Vault Operations Guide. -.

The best time to backup an Enterprise Vault site is after the regular archiving runs that are typically done ightly. After the newly archived files have been backed up the archive bit will be cleared by NetBaCkj. nee the archive bit has been cleared the Storage Service processes the filewatch list freeing space in the es/messages/attachments original locations.

Note: Special considerations should be made when the site is using Microsoft Exchange Journaling, since archiving will be happening continually, not just at specific times of the day.

UStoPPing the specific service for a specific backup is recommended, to allow for quiescent backups. Each Vault Store and Archiving service can be individually stopped and started however only one Storage ervice runs per Enterprise Vault Server. The Storage Service is\the kingpin of the operatIon, since it not duly pro~s Storage SeI vice to the archiVing service but the PST migration processes also. The Storage Service also controls or contributes to the expiry and the filewatch processes. To insure quiescent backups

stop all the services unless you are using the more advanced backup options described later in this whitepaper. The filewatch process is what checks to see when the archive bit has been cleared, so that data blocks can be removed from the primary storage locations.

Enterprise Vault provides a command line process called EVService, with which you can start, stop, or pause an Enterprise Vault service. EVservice.exe is located in the <Program Files>\Enterprise Vault directory. See Appendix B for the EVServicecommand syntax. This can be used in a pre/post backup script stop and start selectively the Enterprise Vault services.

(Within the Enterprise Vault environment it's important to kee the Vault Store database Vault Store

~It10nS, n exes an oppmg 0 ers a m sync. By default the Index directory is shared by all ault

Stores so in practice keepmg them totaIIy in sync for all backups would require that all Vault Stores, SQL databases and the full indexing directory be backed up together while totally quiescent.

·":::(,p:'<Uh'o .:.G(;~~ ~!E,FFA~~· Scitw::;;re <>:':)("aUUl itt !":)his :·!nf·::;·iJ~l ~/EFi"jAS. VEr::'iTi~Y ':>1l.:i,<;"·.' l!':~;: YCFr? r.:3 k:~.::iG, '::·::",(1 nH OH,,,:'f VERlTAS

:~:J~i'~;'l·~:!~;j !l'i·:l(j·'?'rr;~;;d~·:;. ,>f VLF!T/:.,:; ~::C;{iV~'f;;:::~ ;.:,.~, Or:~i:;: ~?'_<:.:U., ";n~<, ,<l~FG'- ,::.~(;:y:;(:~) ,(;(·;ntL)(:fd

::an'iSS and :';'~G:;\~Y::·. t;)'i'- iran·,:rnt·jf1<~, (" i)=-:: tr::;d~:;:~:_'~fk,t~ or ;':'~;:j:~;,t,~;:~~<!

12

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044370

•

-

--

.

VERITAsn,

The first example is of a traditional backup which doesn't use any advanced techniques like online database backups or snapshots. The second example uses the NetBackup SQL Agent for online backups but no snapshots for file system backups. The third example uses the NetBackup SQL Agent and the Advanced Client for file system snapshots. The fourth example uses NetBackup, the NetBackup SQL Agent, Advanced Client, and Storage Foundation for Windows with Flashsnap. These examples are intended to be used asa starting point in crafting a backup and recovery policy for an Enterprise Vault environment.

The backup procedures below assume that OS, application and registry backups are being done and are up to date.

Traditional Backup Methods

U sing traditional backup methods requires that your target data for backup is quiescent. With Enterprise Vault this requires that all the Enterprise Vault services and the SQL database be stopped for the amount of time that the backups are running. Having all the services stopped allows all the data to be backed up and kept totally synchronized between the databases, Vault Stores, Indexes, and Shopping folders. Make sure that the Exchange Information Store and OSlRegistry backups are also current.

A backup needs to include the following:

1. Exchange Information Stores Public and Private

2. File System Agent's managed Volumes ). SharePoint Portal Servers

4. Multiple Vault Stores which each include

a. SQL Database

b. Index

c. Vault Store Partitions Shopping Service Locations Directory Service Database

Master Database (Don't include in Incremental and Transaction Log backups) MSDB Database

OS/registry backups of each server. (System State)

For most customers shutting down their environments for extended periods of time simply isn't acceptable, so advanced backup methods are needed.

Advanced Backup Methods

~..._y.,~~.!::'Jents, Exchange Agents, and Snapshot technologies. These various

backup extensions all "hot" backups r point in time backups so synchronization of the different

Enterprise Vault functIOns IS preserve for simplified restorations when required.

':'>:[)\'T:~;,hi .J)(;;:;: VLnlT!<:~ S:;".;}·.~:::~!"{·) <CQ;(ii>i1 in ;':,j ;;:?',h:·; ,',>}f':: ,_:.,o,:; ~;'fJ:iT/,,~;. ':!CFJ! ~:/f>~! )'/~_:) _k;)C t;:·<~ .(;,H c!l>:,: V~::Frr.l~}:i prc;·:h::) ;';-:.::rr,{.;t3 ~'ind ;;_;;G(it:t'i~: .(-~r,,:~ ;ra(i··:r{;(·'l·k~· or

;d<i:~:'1·:.>t~+j :\ i::(i-::';::;yk,:. :.,1 Vf F,l"rt<:,: ~~Gf:t,'\:!:h;- CG~T'<J(!::tiUL ,:\ ~ >:; U~3 .:t,:i:'·::r cU,-'- ~:,~<" (_ ly:;: ~;,«h<; ::":~:~" ,~;;n::_l:':):' ~"l(;:::;:';f;f; «,r,-)ntGn\:;;:1 i':H(f;,(! (iV,y l)>:·; !X~·:Sbiny<> C: i·i.<~i~~t>:;tt·~:j

13

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044371

1\

backups can also be used for indexes and Vault Store data. Synthetic full backups only need to do

one backup and then each incremental thereafter can be synthesized into a full, which reduces the need

to run weekly full backups.

Backup's without Snapshots

A relatively simple advanced backup scenario can be implemented by using NetBackup and the SQL Agent, without purchasing the Advance Client of Net Backup. This will provide a point in time synchronized backup, and sti allow access to the Enterprise Vault data during a backup, without using snapshot

tee . Illis does reguire that the Enterprise Vault Services be sto.,p,ped in order to place them in Read

illy Mode. While in Read Only Mode all previously managed files/messages/attachments will be

- ......

searchable and retrievable, but no new files/messages/attachments will be managed until the Storage Service

. s turned to the Read Write Mode.

Required components:

1) NetBackup 5.1

2) Netlsackup SQL Agent

3) Enterprise Vault v5.0 SP2

4) Enterprise Vault Operation Guide

(This has been tested using Enterprise Vault v5.0 SP2, NBU 5.1, and the NBU SQL Agent.)

i

\

~-

By using the instructions in the Enterprise Vault Operations Guide, setup the registry scripts which will switch the Enterprise Vault services into read only mode.

(-----:_

i 1. Create registry entries that enable Read Only access to the Enterprise Vault services

2. Create a registry script to automate the read only registry changes.

3. Create a registry script to automate the read/write mode registry changes.

4. Create a Windows batch file to execute the registry changes to read only mode before the backup starts and return them to read/write mode at the end of the backup. (See Appendix A for the example

batch file backup_ev.bat) "l

5. Create two SQL backup policies, one for incremental backups and one for full backuPs~. ~e' that the full backup includes the s~ecific Vault Store ~atabase(s), Directory ~atabas~' S. DB .. 'land

Master Database. (Make sure the incremental/transaction log backups don't include easter,

because SQL doesn't allow incremental backups of the Master Database.) See the NB QL Agent

manual for details on setting up the SQL bch script file. Example bch files are included in Appendix C.

6. Create a backup policy with both incremental and full schedules that includes the Vault Store Partition(s), Indexing directory and Shopping folder.

Considerations:

o Multiple Vault Store Partitions can exist per Vault Store.

o Make sure that if separate Index directories are defined that each one is backed up.

:(·H.".:::~",·;;'(i 'jET::iL"~;~;. -V(';;:;;iT/,_S .~/-fLf>'~':<·'~' ~>t·_: VFF!T;"\~; ~;;:':(! Hi! (»~::,f VEF-UTfd prG,jt;:~i ::'::.dr;f;~) ~~n;j S.jt);;FY~~~ ~·lr~- ;r(:,(j·:>rn{·:rx~~ <':

,./:<;:.~ :::<;;J;;:;;~'~; cnv:" ~x>:l'.;u·r;8:'><·, )GX:;(:~; f(;~·::nli{)(;H·j k;(er: (t;t;y i)('; t(f:,d6::\'lfi'[) V ·:<y:;U:£~:;:j

14

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044372

,

-

, ~

VERITASTM

o To prevent the file system backup policy from starting automatically by the NetBackup scheduler, make sure that each schedule is defined but doesn't have any open window that it can run in. This will allow the batch file to start the backups AFTER the services have been stopped and started in Read Only mode. See Appendix A for an example of using the

V\ Windows Scheduler.

V Make sure that any other backup of the Enterprise Vault Index, Vault Stores, and Shopping folder is prohibited from occurring. Setup NetBackup client exclusion rules as needed.

7. A system OS and application backup must also be performed but is assumed to be configured separately from this discussion.

Reference the example backup_ev.bat file in Appendix A on how to coordinate the backup sets. The backup_ev.bat syntax is <pathname>\backup_ev (Full I full I Incr I incr)

See Appendix C example screen shots ofNBU policies, including SQL Agent backup scripts.

NOTE: Netliackupprovides a bpstart notify and bpend notify commands which allow batch files to be run at the beginning and ending of a backup policy. These policies are not illustrated because of the complexities of calling a non-file system backup from within a file system policy ( bpstart _ notify and bpend _ notify are not supported in a SQL policy), and some of the idiosyncrasies that would have to be overcome with a more advanced batch file.

Utilizing the NBU and SQL Agent requires that the Enterprise Vault environment be kept in a read only mode until all the backups are complete. While this may not be an issue to some customers others specifically those implementing the Microsoft Exchange Joumaling feature may need a more advanced solution like the following.

<~(·p~·'~')h! .J.):.Y·~ VH~rr:·<~ Sc~t;!/,:j:rG C::·q:.:=::: :.:{!( " ;Dji ::yU) :-;.;~}~~~.:.;,,:'(j v{;:qr-,~\?:;: '=ffJt'[ l,Y ~;:Ah'n:': ihe ~JCFdTA~~ !(;)(, en·:! .(iii O}'k':' VERrr;:.,s prc:Ju'-) i>i;T,(,,'), :;~n:j s.jG~r-E!~: EH"';" ;r;.H~:':!!T;'[·';;·~-:" ('r

F-:<(J",t':::<G'l :r:~~(j·?r(:;;:;:-k':, <,f Vt:.FTLt<::; ~:~uth·':<~-: ~:!~': ::':i dfU!:"= ::U.~ ~-:><-, .' ~? ,:<iLl :. r(;r>:'(; ·':In(F:)( '~\/:::::';r:;;; !(;(:t,licor,G·j h(-;((-;l(, ((:[."y >:; ;lr.~j(·;n;y;·<., C( :'-·:-0;~'t>:;"·~!rl

15

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044373

" -

.. ~

VERI

Backups with Windows 2003 Volume Shadow Copy (VSS) ~ , ~

b~~-v-/

Requ,ired components: ~ f\ j • ~,V

1) NetBackup 5.1 /;;;

2) Windows 2003 Servers

3) NetBackup SQL Agent ' ,~

4) NetBackup Advance Client ~

5) Enterprise Vault v5.0 SP2

6) Enterprise Vault Operation Guide

By purchasing the Advanced Client components of NetBackup, snapshot techniques can be implemented

which will reduce the amount of time the Enterprise Vault services need to . ead

~Y mode, As tea a ase, a ore artitions, and Indexes grow, the length of time required to back

them up will also increase, sna_pshot

--

1:s:(chnologiesallow for a point in time

backups even if the actual backup tiluetakes hours. By utilizing NetBackup ~th Adva'i1ced Client, backups can be configured which interact with various snapshot providers to automatically quiescence a volume, snapshot it and then while it's being backed up the application (Enterprise Vault) continues as ifthe backup wasn't happening. At the end of the backup, the backup image will be as the volume was at the time the snapshot was taken. This allows multiple volumes to be backed up insync even thou the backups finish at

. erent ti es. To create a snapshot the volume needs to be quiescent, with Enterprise Vault v5.0 this requires that the services be stopped. But unlike the advanced backup without snapshots, the services don't need to be put into read

_gnly mo&: The services just need to be stopped, snapshots started and then

'--I=4~lfIed. The example batch file backup _ ev _ vss.bat in Appendix A shows how this can be done. Within NetBackup, it's as simple as enabling the Perform snapshot backups checkbox. Under the Advanced Snapshot Options dialog box are four options, auto, VSP, VSS and VxVM (if Storage

~l Attriputes I ~5chedulesl ~Cliel)llr~Backup'selectionsl:

Change Policy - Inde!l~¥ault_Stofe 13

~"-"--""'''''-'''''''''''''''''''''''-'''-~-- C""~"';'-';""";"__;_.""",,,;_~"-'---

_ Destination _. ----~+~:,I Lr-r- ,:~~~~r::'D7 PMdi

, " I' 1, ' ' H"Tkup !:H)(\Ncrk rkif""'

Policy storage ",nit: ••• ..... ...,) •• /

li2lsr2c-disk ' Ell i

~:~~:a~:I~~e pool: , :£Ill

.:.3

-_ ,:: ": .", .

r Collect YL!e image restore information

[T,' :,,~~~~th {~'lt,!~-. deA,edjtr; .. - :

r (Required for synthetic backups) , ": I

, ,Compression

, r EncIl'ption .'

I r kollect disaster recovery information " ! I ! P' Allow muitiple :Qala streams

I

i r CheckpQint restart every:

, Ii] ':H 'unVYt

, p, bimil jabs per policy:

: 12 ':H

·<;::,)yrqh .::D:_/; vr.r-;;(ri·~~:: S(lit·.$;~~r-G '"C'C<,",""""" C<'),",,,,,,,"(),j "BC"',,,",+:, < VEFiT,'<,

16

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044374

Foundation for Windows is installed). By not selecting anything in this pull down or by selecting Auto, NetBackup will pick depending on what snapshot providers are available and configured. If a particular snapshot is desired then select it in the pull down dialog box.

Microsoft Windows 2003 provides the Volume Shadow Copy Snapshot(VSS) and NetBackup automates the

creation of the snapshots. '

The following example of an advanced client backup shows the use ofVSS.

The process is different from the backup_ev.bat process in several key areas: First, the services are not restarted in read only mode, thus the Storage Service doesn't have to be stopped again at the completion of the backups. Second, the batch file processing doesn't have to wait for the backups to complete. This allows resumption of normal processing within the Enterprise Vault environment. By using the snapshot technologies more advanced restoration methods of SQL can be used. Therefore it is also recommended that a NetBackup policy be created to backup the SQL transaction logs. This will allow for recoveries of the SQL databases to be much more granular than to the last full or differential backup shown in the previous examples. When recovering a SQL database with transaction logs, first restore the latest full, and then the tratU:6lojon logs to the specific time of the last backup ofthe Index, Vault Store Partitions, and

..J>hopping folders. This allows the database backups to be scheduled at different times than the EnterprISe

, Vault volumes and still provides a method for keeping them in-sync if a recovery is needed. This can reduce the total time of the Enterprise.Vault backups, since it now only needs to have the services stopped long enough for snap sbots to be t6lken of the volumes that contain the Indexes, Shol?Ping folders and Valllt. Storag~rtiti~. The NetBackup SQL Agent utilizes the SQL VDI interface to create a snapshot the SQL

( databases.

Another advantage is that the SQL full database can be taken less often if desired and the transaction log backup more often and still allow for recoveries of the SQL databases.

To enable transaction log backups ensure that the SQL databases are set to Full mode verses Simple or Bulk-logged.

1) Start the SQL Enterprise Manager.

2) Open the databases and right click on each of the databases and select properties

3) Select the Options tab

4) Verify that the Recovery Model is set to Full.

a. Make sure that the Master and MSDB are also set to Full

The next step is to create a Transaction Log backup policy in NetBackup.

1) From the Windows Start, select programs, then VERITAS NetBackup, then NetBackup Agents, then NetBackup MS SQL Client.

a. Select Actions and then Backup and you will see a screen similar to the one below.

b. Select the Backup transaction log (database only)

c. Select all of the appropriate databases in the Enterprise Vault environment, making sure that msdb is iRduded...(Exc1ude the Master from the transaction log backup policy)---

. '-

/>-'~)~:~\/:i ,'0<-; ~;L?iT/<:'. ~>:;iL~:~:;.!-F; {::::"r.:(":: 1::.:tf( , }\~ ::/:~3 'f.-}';,,;·,'y"··:<l VEH1T/.':;. "VCE!T_t,5 :;3:::f'; 'j; (.",",:.- :: :~., '.! :d i);~~

/, ,",::i ,,..;f., f' '~" '., ~ ~ . l"\, ' ,~< _"" :,.

1, ·;';d·:·;~:·,.e;:·j-.:~ ,~-t \~',e:: :-~'.':: ;,:::,'~':j;"f';j ("(~:-::r:;<-~(,;!:;:.:.; "" ",' ',C;'''.'" ; ••

)[0')(;') :\iVUS ;"1!,!j 3jDQans ~.}!,,<:. ira:::1·:,.r(inri{,,· (=;' hG(<::';;(' tri.:.;;,: D.::·; t(:::df;::';drl'"L U( ;":}{ji::,t~';:H(~

17

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044375

~ ~ ~

VERITAS'"

d. Select the save for later execution

e. Selectthe. NetBackup Policy, click OK, name the file and save it.

Backup Microsoft 5Ql5erver Objects I!!I~D

......

/

...... : : 1 I SQL Transaction

appearirlg In the righthand panel would you like to back up? ·········1

I· (0' onI . . .selected .«: All, except those selected r All I I •• '. OK. • •••• 1 C;lncel

L":_'-"_. __ ~c._ ~_ ~ _ ~~ _ .1

DBMS Instance: SR2C\<deFaIJIt>

SeJect database{ sHor bacl<i.Jp ance. <

SR2C\ <default>, . .'

.r-~~~~~----~-----------

Enlerpnse',I,ultDltector •

.E\,'FlleArchl· e

Evjournel

EVVa1J1t5tore

master . model

: Northwind pubs

i"Backup type: .. -- +: =r=»: ..

I r . Full backup

1 :i :::t~1~:::~~~\::c;~~::::b::;ndfilegrOup: ;.

I .··IBack up and t;~nc~te tran~action log '3

, Backup script ..

How many Microsoft SQl Server stripes? rr- ~

. · ··....1

I

.. .I

r: Launch inunediately

NetBackup policy: •

The example backup _ ev _ vss. bat in Appendix A shows a batch file that stops. the appropriate Enterprise Vault Services (each site will be different as to which services need to be stopped and started). The batch file calls the bpbackup command ofNetBackup which starts the specified backup policy and schedule. The NetBackup policy has the Advanced Client option, and the Perform snapshot backups check boxes selected. Additionally under the Advance options pull down box, select VSS. This specifies Windows 2003 Volume Shadow Copies Snapshots.

In order to make sure that the snapshots are done before restarting the Storage and Index services the example batch file backup _ ev _ vss. bat has a sleep 60 command. The sleep.exe is part of the Microsoft tools that can be downloaded. The sleep command sleeps for the amount of time specified, in this case 60 seconds. (The Microsoft Tools URL is: http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69- 57ff-4ae7-96ee-b 18c4 790cffd&displaylang=en)

Each site may need to have different timeout values depending on how fast the actual snapshots take to

(" ~reate. ( ~e batch file could be made to watch for the successful completio t e snapshots in the

\._5Program Files>\VERITAS\NetBac up\logs\ p IS <date>.og 1 e but for this example it's not done.)

To verify that the snapshots have completed before the batch file restarts the Storage, Indexing, and Shopping services to the following.

·<»)j.-,!\?r:i .:,'[)(i.:) VEHiT/,::: Sdt";.",·S,fn i.>.:y,,-:.~~k;:·~ /:...,~; :,/"i:.; (G'k: ":<~ ··.iCFTL~'.:'::. \/FFUTl·Y ~>fi"_',·{·,,:

;:.)-(;:~>.:>:::;.j ~"i.<';". (;::·:~k·:· J VEHTf<:; ~~:::[i.,;\"!.~:~.)- G(;,(;G:'(:';tiG( " ;';~:: \ !~. x:d,·:':; "'::':;"<', :.,'1,>:: u~'n -."

(,,··~{.h~nfFi_~; A ' ,(>,' i\i'5:pe{:'~i'/': ~'~<~ :j"~~ ::::",_'<', ~ "': " c- pi' -:.<~: <1: -/~:::~:'~::;;~;;<';' )"\.:_,i'( ,:1"[ .. =r.j(' '·//-f·:,; it

:'''-,(= <X·):;-'! VERlTA~; Ofu;:L-) ;:-:;'T(lG;'3 ~~n~:i :;;iG:J:':~(i:'~ {.':r':~ :ra(j··;rr/Y},:·;~ ,:.,

'::i:T~'1f;b ;r!f·.:~~fiU(;H·j ho(:(·;,(; ((it,)"" ~):::,i.(:.·:,k1::"Yt<,;, Gi ;':'<i':")t;::;,;HD

18

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044376

.. ~ ~

VE RITASTM

1. Create the N etBackup log directory for bpfis.

a. Either run the <Program Files>\VERITAS\NetBackup\logs\mklogdir.bat which will make all the NetBackup logs or just create the bpfis directory in the NetBackup logs directory.

2. Create the NetBackup policy for backups of the Enterprise Vault volumes that are to be backed up.

For example v:\Enterprise Vault Store\Vault Store Ptn1, <Program Files>\Enterprise Vault\Indexing, and <Program Files>\Enterprise Vault\Shopping.

a. Make sure that Perform snapshot backups is selected in the NetBackup policy

b. Make sure that in the Advanced option pull down window VSS is selected.

c. Create both a full and incremental schedule for the NetBackup policy, but don't configure any open window for NetBackup to start the policy. The policy will be started from the Windows Scheduler and the backup_ev_vss batch file.

3. Customize the backup_ev_vss.bat file to stop and start the appropriate services for the Enterprise Vault environment.

4. From the DOS command line, run the "backup _ ev _ vss full" which will run the full backup.

5. After the completion of the backup, open the Windows Event viewer.

a. In the application look to see what time the Enterprise Vault Index and Storage Service restarted.

0<01)
@J_iot"" o .... E .....
. ~SO<"' .. 12129/2004 Enterprn;eo"'.;,uit St<.lr<llQe Servet' 6221
. Sy,,","
~ DNSServm ~lriorTMth:«1 12.fZ9l.!004 10:42:17 AM VERJTA'S vss Provido!-. N~~ 11 N/A SO'.zc
$InfOtlTj~ lzr.t9/2004 10:42:17 AI'''l VEP.1TA5Io'5S PrOVIde! ''''''' 11 N1A SP2C
! J;J Info"".tjon lZl2.9/ZOO4 10:42:(1 AM VER!TA'S YSS Pn: .. vtder None l! N1A SIU<
'~W""""'ion - 12/291200'1 lO:"'1Z:11AM Er:t1Y.prfleVN.,;t Shccpp:ing 5ervke 5156 N/A SRZC
Q) lnfo'l'fl~ion 1$912004 10:042.:11 AM v ERIT AS VSS Pro",id.." N:.,IT!Iit 11 NiA S!>zc
{.v'Ir/Ol1T~ 1.2/:JW2rJ04 1Lt:42:1CAM Er.l;erpuse Vz.Ut Jo.I.lt'nai Servsce :<160 N/A SP2C
:» Irlormation 12/2.9/2004 . 10:42:09 AM B'ot:etp.i$O:lIV",wlt J\)f_"r~5eoNk.e 2213 N/A SIU<:
(!), l.nformatlon lZ/Z9,t200oJ 10:4Z:00-A-M Er.t~ev,~ Reb:ieval se-we 2,19 NIA SRZC
(!)tnfomwstioo 1:!/29f2004 lOl42:07AM Entctwp~is.e\.'auIt: j:(:etrievalS~·.1'e 2213 N/I>. S!>ZC
Z!)IriotrtJ&ioI"l 12}29/2))04 10:42:Q6AM f.'J':t~ril:ev~ iOJ<;hjve serwee 2'173 "iA snzc
';::0 Inf<Jrm$ion 12/~/zoo.\ 10:42:04 AM Erot-..erpris,etV'!PA An.:ruvt': Set'viv~ .2'02'13 N,fA SR.."'<:
(.I.) information 12/29f2OO1 lO:"2:00Al'4 Er'tc<"Pl'i$eV~ lrode..,ero~ 7169 NfA ... ac
C!) Infonn<!ltion 1:!129fZOO-4 lOl-H:53AM El\tup!'i.,;;e ..... ault: S!::ofage$o;;r"C't 6ZZ2 N/A SR2C
(~tnfotm~1 12t'.t9!2tKH 10:41:,,0A."1 E'_"t~p'ise v~;lI:: $hop)::4n9 servee 5107 N/A sszc
<yIoform~ 12i~/200" 10:41:33 Ao"'l Ent:e<'l:)f'~e '.J~ JotJrnal~rvke .2183 HIA SRZ<:
Q.)W01mation 12/29.r:t:OO'f to:41:3QAM f.nt~prl'"~ y&uIt R~h·iev.!ll SerVICe 2:132 NfA SR2i:
(A) tnkllmabon 1;:fZ9/2004 10:";1:27 AM Ent~'Pris:e y",u#: A~(.hivt) 5<e1Vi~<!I 21S1 N/A SRZC .................... , .....

!

· .. ·· .. · .... -r .. ·'· .....

Windows Event Viewer

6. Compare this time with the <Program Files>\VERITAS\NetBackup\logs\bpfis\<date>.log file. a. Look for the start of the start of the NetBackup policy (Example below).

10:42:08.783 [5920. 7768J <2> logparams: C:\Program Files \ VERITASWetBackup\bin\bpjis. exe create - nbu -owner NBU -id sr2c_1l04338527 -bpstart to 300 -bpend_to 300 -backup _copy 0 -ru root -pt 13 -rg other -fim auto -class Index_Vault_Store -sched Full -fso -S sr2c

.J::<;~;~·'(iJhf /)(;5 'JEenAS ,:::/:it~">'~lm <>:·:i>U::1k.',) ,t',ij ~HT':.;·~~n.;.,i ~((<·'::-i((·;d<~~.~ (',f VEF!T/<~~ Sd('f~:;''';'ui;';;{~:;;fi'i<;; .S ~A it;f"':; '··:;:sp:;?(;i,·x,: ~:~·'tr:~;.;:~ ,~!;:~: ';0",,·;{!";·"··;n;

r.::Xi nH ()ji"'h'·:: ,:/[:J<.rr.<':._s )!'c..:hi·) "artiC",; :·:;nd '.>If)q;:.<i<~ ~.:~"<{~ jr;.<j,,~f!'<::'Y-;' c.r ~jV:i3(::~ ~r:Gr int:<':;') ku·;:(; (t::;,·~: i,)..:; ~(,=:-,df..;?:·:,A~S (;i' ,,:~qi<:,[ee·.~i

19

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044377

" 1 ~

VE RITASTM

b. Look for the successful completion of the snapshot.

10:42:30.596 [5920. 7768] <4> bpfis: INF - EXIT STATUS 0: the requested operation was successfully completed

c. Compare the time that the Event logs has with the pbfis log, and make sure that the pbfis log shows that the snapshot completed before the Index and Storage services started again.

d. If the services started before the snapshot completed add more time to the timeout until the . snapshot completes before the services start.

Note this only shows that the snapshots completed not the whole backup

7. Verify that the NetBackup Policy completed without errors

8. Periodically check to make sure the snapshots are completing before the services restart.

Note: A more fool proof method of making sure the snapshots complete would be either to create a more elaborate batch file that watches the bpfis log, or use the VERITAS Storage Foundationfor Windows and use the Vx VM snapshots that can be scripted to wait for the snapshot to successfully complete before restarting the Enterprise Vault services.

Backups using NetBackup Advanced Client, Snapshot techniques and VERITAS Storage Foundation for Windows.

Another possible approach is to use the VERITAS Storage Foundation for Windows product which includes the VERIT AS Volume Manager. Due to configuration limitations and time constraints for this document this particular configuration has not yet been implemented and tested. Never the less, based on the testing done with Enterprise Vault so far the following configuration should work without any problems.

The components required are:

1) NetBackup 5.1

2) NetBackup SQL Agent

3) NBU Advanced Client

4) Enterprise Vault 5.0 SP2

5) Enterprise Vault Operations Guide

6) Storage Foundation for Windows 4.3

7) Flashsnap Agent

8) SQL Books Online

9) Must be running Windows 2003.

Storage Foundation for Windows provides many advantages for managing and maintaining a Windows environment. The features of diskgroups, mirroring of volumes, and Flashsnap allows NetBackup to automatically split a diskgroup mirror, deport it, import it to another machine and back it up "off-host". By doing this the impact of backing up the Index, Shopping folder, SQL databases and Vault Stores, can be

..(>::·P/;~~h1.::DCb VLFFf<:; S(:1tN~:.:.r{·:: C:~·:-)(n~tk:'::·l re(;:~~~·-:-:r-;·j ~rf.~(;·.-:';(:-~·H·k~;.:>l VLFjTj.~~~ :;GfL~:':"i:::·:- i:,{,,:c,,"'''' l''':;di:';:rn,.~r~.s :.A t~';(-~:r i"is:,~~:.;::;d:·:t <_"(~:',:){~:·:;~:~;S

20

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044378

;:;.. ~

~ T

VERITASTM

taken away from the Enterprise Vault server, and SQL server. When the backups are complete, Storage Foundation for Windows can snapback the volumes and re-sync any new data that was created during the time the mirror was split. Once configured all ofthe splitting, deporting, importing, backing up, reimporting and re-syncing can all be done within a NetBackup policy.

Note: Additionally technologies like the VERITAS Volume Replicator (VVR) can be used to replicate the data from one physical location to another to help in the disaster recovery and site failover protection needed by many VERITAS Enterprise Vault customers. NetBackup also has synthetic full backup capabilities which allow one full backup to be taken and then incremental backups from then on. With the combination of synthetic full backups and disk staging of backups the Vault Stores, Indexes, and Shopping folders could easily be configured to use synthetic full backups which will greatly reduce the backup times while still providing a quick and easy recovery.

Recovering Enterprise Vault

Backups are done in leisure while restores are done in a panic. Author unknown

The first step is to determine what needs to be restored. Refer to the Enterprise Administration Guide and Operations Guide, Disaster Recovery section for how to recover and the order in which to recover them.

One interesting fail-safe procedure that is available is the ability for the Enterprise Vault to rollback archived messages and attachments that haven't had their archive bit cleared yet. What this allows the administrator to do is un-manage the messages and attachments that have been archived, but not freed yet from their original location in the Information Store. If a Vault Store Partition is damaged, the administrator can "undo" all the pending archived messages, attachments and files, since the last backup of the Vault Store.

First restore the latest full and incremental backups of the effected Vault Store. The procedure below allows the administrator to cancel the pending files that had been copied but haven't had their archive bit's cleared yet. (E.g. Not backed up yet) These files will still be on the filewatch list. Once these files are "rolled" back they will be eligible to be archived again during the next archive pass, as if they had never been archived. This is the reason the safety copy feature exists!

Note: The PendingShortcutTimeout only applies to Exchange data no file system or SPS data.

Create a registry entry on the Enterprise Vault Server that controls the Vault Store Partition that is being recovered. A registry value is used to trigger the services to cancel archive pending items. The archived date of any archive pending item is checked against a number of days specified by the registry value. If the date is older than the number of days then an archive pending item is cancelled.

The registry value is called:

PendingShortcutTimeout

·::~'::)/ri)rjt .::{)Y-:; \;LF~iTf-\D S()1tH.:lt~·) <:c'!;cn~~iG:': ,~:q'Y·::(G·j '~aC~:'{;~:/~"~ ,::f VLF:1T;'<:; ?Ch:'k!:::;:) ir··;j(.1;~C":~~c·l:'.:" :A l["if\f i·0_::f;(-~r~i:·:c (.-c,r:;";t~,·:/~:s """""""""",,',,

! V::F_iT_.~\:~;. ·::./rY~T;\;:~ S:At'~·[·1(<:.: j!'k; \fEn. i';"\~; \}1(: ;:.".,(1 {·";;1 Gil>:", \!CFdTt~.S y"(;·:1~;·~! ,"ur(,:f'>} :·~n:j sio~;am:, ~·:r<,:~ ~i"~:<~··:rr:':;lrY~< or

t 'S ,1/',11:::' u; '::r:.:<~ C;f1":,; (,'-):':,~. (; .,.-',( ,;, ~.l::;~,)~·::u; ;r;~·;niit)(;G,_j h('-h';'!" ;-rlti";' ~y, t(~·:d~·;;:'.:l(l'!~ cr: ,:.>q6t>:·_':H(1

21

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044379