BACKUP PLAN

"-, ,.'_ , .. -.'.. , .. ' , _._ _ _-'.", - ,

THE EXECUTIVE OFFICE OF THE PRESIDENT OFFICE OF ADMIN ISTRA TION

OFFICE OF THE CHIEF INFORMATION OFFICER

MARCH 27, 2006

DRAFT - EOP-OA-OCIO-A&E

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044382

ECRMS - BACKUP PLAN

DOCUMENT INFORMATION

DOCUMENT REVISION HISTORY

MARCH 27,2006

DRAFT - EOP-OA-OCIO-A&E_

PAGE: i

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044383

ECRMS BACKUP PLAN

T ABLE OF CONTENTS

1.0 Background 3

2.0 Backup Plan ...........................................................•.........................................•................. 4

2.1 Backup Attributes 7

2.2 Maintenance Windows 8

2.3 Backup Considerations 10

3.0 Restore Procedure 11

4.0 Backup/Restore Tests 12

Appendix A - Checklists for Backup/Restore Tests 13

Appendix B - System and Application Files Backed Up 13

DRAFT - EOP-OA-OCIO-A&E

MARCH 27, 2006

PAGE: ii

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044384

ECRMS BACKUP PLAN

ECRMS BackuplRestore Plan

1.0 BACKGROUND re..(014 ~t 1\1

:sqJtew\ "1

There are three sets of files requiring backup that are associated with the ECRMS s~n. '

These are:

• Source PST files that exist on_

• System files and ECRMS application files that exist on the ECRMS servers

• ECRMS data

Eo{) lOA/ 0(10 IIStT

Both the source PST files located on SFEOPOI and the ~stem and application files located on the ECRMS servers are backed up as part of the standarflS&T backup process. This process extends over a 14-day period, with a full backup, incremental b ups, and a level, I rollup followed by additional incremental backups. For reference, is of the ECRMS server' directories that are included in the standard IS&T backup and a provided in Appendix A.

tJ.

The focus of this document is the backup and restore of the ECRMS data. With ECRMS being

the integration of three COTS products, MDY's FileSurf, Symantec's Enterprise Vault and

EMC's Centera (content addressed storage), there are multiple sources of application-specific

data requiring backup. These are:

• FileSurf SQL Database and Transaction Logs

• Enterprise Vault SQL Databases (Directory and Vault Store) and Transaction Logs

• Enterprise Vault Full Text Indexes

• Enterprise Vault Archived Emails (stored on EMC Centera)

The main requirement of EOP for this backup and restore process is data integrity. The integrity of the data after a restore is much more consequential to EOP than, for example, the time it takes to perform the restore itself. Regarding the backup and restore timeframes:

• The goal is to minimize the backup window in order to maximize the ECRMS operational time, specifically to import the daily emails and legacy emails. There is a large backlog of legacy emails, so the more time that can be allocated to the import of these emails, the shorter the timeframe to complete the processing of the entire backlog. However, this goal cannot compromise the data integrity requirement.

h . ~\f\~~a h data i h lower nri h h'· . f h d d

• T e time to restore t e ata IS a muc ower pnonty t an t e mtegnty 0 t e ata an

reducing the backup window. It is acceptable to EOP if the data restore takes a couple days. This is the case since the number u ers: f the system is very small and the need to interact with the system (for searches, expo~ and rrco ""management) is not on a consistent basis.

(lUC(llln,,\ \t;C>Ik QO;I\\ ~,ck2.

- 0\ - W I'iJ. ~ l ~ raJiu-._ ...

MARCH 27, 2006 - ECRMS BACKUP AND RESTORE PLAN \QACV- ~~ \_(J2j{\S PAGE:

\L' .... !'It \ lit I IN'." ~ 1 I a)j TiN w.R,"J\.l")

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044385

is Wfti t r(,\'\ll~ . Note that the backup of the source PST files (on SFEOPO allows a second lelel of recovery in the event that there is all, iS6ye with the ECRMS data b kup/restore proces! The source PST

files can be reprocessed from wherever the ECRMS data . This means taking the time to] (cvlu51~

process PST files that were processed in the past, but in some circumstances, this may be the ~ Wi M4Ih,~

better alternative. )

~\(itJ .. ~ fto;tS;fl;6/i: dt ~(}Q\~* Atct(I~~1 (d(tlJ..utQ ',h btl 1 ~v7',J. !ollis 1/

In addition to EOP's backup 0 the following backup constraint is imposed by Enterprise

Vault and must be addressed b ECRMS Backup plan:

ECRMS BACKUP PLAN

• Enterprise Vault data must be backed up in a static and synchronized state:

o The data includes the SQL databases, (full text) indexes and the archived data;

o The Enterprise Vault services must be either stopped during backup, or alternatively stopped, then placed in read only mode for the duration of the

backup, then placed into read/write mode and restarting of the services. . .,

LoE wla.t <).v\\l\(v!.lr4Y,~YtlISWY\

o All backups must occur within the backup window. ' • .A • ~ L __ " QMI\.~.I 1

f.... wft,..;.. dor.. V) W 1\ WI \1..1 .... ,1'.-1 t VI' I VW\ •

In regards to backing up the ~~ data (i.e.: the emails themselves that are stored on EMC It ~c\ 1

Centera), EOP has chosen to r~the Centera to a secondary Centera.

1 .

. . . • . W' f'>(f{l.a leMlj'- WLGtwtct::;_ ~j

As additional background, below IS a st of all the ECRMS databases: 1"- \. .

t&l 1- o\((~ ttc.\ ~~ WP-i\ i

• FileSurf"ECRMSDB" database .L:''\.J (JA.t llYt 1"Iy}u

This. database includes metadata and records management information on every~ t" J t.M ~~ ..

~. M

• Enterprise Vault "Directory Service" database

This database holds all the Enterprise Vault configuration information.

• Enterprise Vault "Vault Store" database e

This database contains information on each chived e. ail, including storage information as to its location on the Centera.

2.0 BACKUP PLAN

The Backup Plan for the ECRMS data involves:

1. Replication of the ECRM archiv emails

2. Backup of the SQL transacti gs via regularly scheduled SQL Maintenance Plans

3. Backup of the ECRMS databases using SQL Server

4. Backup of the ECRMS full text indexes by copying these files to another disk location.

5. The files resulting from steps 2, 3 and 4 will be written to tape as part of the standard IS&T backup procedure.

MARCH 27, 2006 - ECRMS BACKUP AND RESTORE PLAN

PAGE.

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044386

j. ECRMS BACKUP PLAN

h (N'e Mlv.M In tc\

I - Replication of the ECRMS rchive em~ccur between the Primary C ~tera and a secondary offsite Centera. Th repji tio'U be scheduled to occur during certt: time periods.

2 - SQL Maintenance Plans will be created to back up the transaction logs for the ECRMS databases. During ECRMS operational hours, the FileSurfDatabase (ECRMSDB) logs will be backed up hourly, while the Enterprise Vault Database (Directory and Vault Store) logs will be backed up every 3 hours. The SQL Maintenance Plans will also be set to delete transaction logs older than 2 weeks.

3 & 4 - A master script will be developed to control the backup of the ECRMS databases and full text indexes, since these must occur 'in sync' and only when the ECRMS services are in a quiescent state. The master script will stop all the ECRMS services, then back up the ECRMS

data from the various sources and then restart the services.}J., .

. /: W(l<t{4 lki~ ~ a. lIb I, S£II(,tu_r rur, dV' (l,\'n (Oh(i.i,W~t~ Two versions of the master sc¢t will be created, one for incremental backups and one for full 1f-sl backups and database Ch~~S:\f. he incremental backup script can be ru.n daily (5 days aweek) . and the full backup run ~ however, the schedule can be tailored as sui1EOP needs. The

differences between the two master scripts are listed below. -to

The Incremental Backup Script will perform:

o incremental backups of the ECRMS databases and full text indexes.

The Full Backup Script will perform:

o full backups of the ECRMS databases and full text indexes

o a database integrity check (DBCC) on the ECRMS databases (after the full backups)

o a full backup ofthe FileSurfand Enterprise Vault system databases (i.e.: master, model and msdb). The master, model and msdb backups will be placed into one file for FileSurf system databases and one file for Enterprise Vault system databases.

The master scripts will stop the FileSurf services first, since FileSurf is a client of Enterprise Vault. There is then a delay to give the FileSurf services ample time to shut down, particularly if they are in the middle of processing a very large email. After the delay, the Enterprise Vault services will be stopped. Then the data backups will occur, after which the Enterprise Vault services will be started and finally the FileSurf services will be started.

The master scripts will execute on _under the dsxs-ecrms login. Remote Service Management to make remote calls to _and backup activity across all 3 servers. Regarding the database ~""-''''''''P'''''''''''V

--:;:'---..----

Maintenance Plans for these backups and for the scripts to initiate these plans

uh II ~ l0'ko.:t t\ar~ If elCfl1 lSi1rl

fdl\omcl !

The following diagram depicts the overall flow of the Master Scripts (Incremental and Full).

MARCH 27, 2006 - ECRMS BACKUP AND RESTORE PLAN

PAGE:

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044387

ECRMS BACKUP PLAN

ECRMS Backup Flow

Stop

FileSurf Exchange Services

Delay x minutes

Backup FileSurf ECRMSDB Database

, J _

, ,

: . cri tOnly: , : PerfO~BC on FileSurf : , ECRMS atabase :

L~~~~~~r------:

1-------- --------1

, ,

: Full Script Only: :

, Backup ,

: FileSurf System Databases :

L :

Backup

(Full Text) Indexes

Stop Enterprise Vault Services

Backup

EV Directory and Vault Databases

,-------*-------

I I

I Full Script Only: :

: Perform DBCC on EV I I Directory and Vault : : Databases '

'------- -------:

Start Enterprise Vault Services

Start

FileSurf Exchange Services

, J ,

, ,

: Full Script Only: :

I Backup ,

: EV System Databases :

I ,

'- - 1

Diagram Notes:

The backups (databases and indexes) are either 'incremental' or 'full', based on the script being run.

The dash lines indicate additional steps in the Full Backup script.

The scripts can accomplish the above in a serial manner, as long as it does not significantly impact the backup time.

MARCH 27, 2006 - ECRMS BACKUP AND RESTORE PLAN

GEORGE W. BUSH PRESIDENTIAL RECORD

PAGEl

OAP00044388

ECRMS BACKUP PLAN

2.1 Backup Attributes

The following table lists -the attributes of the ECRMS data to be backed up, the source location, the planned frequency of the backups, how the data is backed up, what controls the backup and finally the location of the backup files.

Data Location of Backup Backup Backup Location of
Data Frequency Mechanis Controlled Backup
(can be m by Files
FileSurfECRMSDB SQL Master
Database Backup Script
FileSurfECRMSDB SQL SQL
Transaction Logs Hourly* Backup Maintenanc
ePlan
FileSurf System - Sunday Full
Databases

Enterprise Vault - Tues to Sat
Databases (Directory Incremental
and Vault) - Sunday Full
Enterprise Vault - Mon to Sat SQL SQL
Transaction Logs Every 3 Backup Maintenanc
(for Directory & Hrs* ePlan
Vault DBs)
Enterprise Vault - Sunday Full SQL Master
System Databases Backup Script
Enterprise Vault Full - Tues to Sat File Master
Text Indexes Incremental System Script
- Sunday Copy P
Full * *
Continuous EMC Centera COOP Site
EMC or Scheduled Centera Replication
Centera Replication* Replication Schedule
**
* The Transaction Logs will only be backed up during ECRMS Operational Hours.
See the schedule in Section 2.2.
** As the index space grows, the full backup may be performed monthly instead of weekly. MARCH 27, 2000 - ECRMS BACKUP AND RESTORE PLAN

PAGE:

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044389

EeRMS - BACKUP PLAN

Sunday

Monday

Tuesday Wednesday Thursday

Friday

Saturday

12AM

4AM

Maintenance Window (Incremental Backup)

Maintenance Window (Incremental Backup)

Maintenance Window (Incremental Backup)

Maintenance Window (Incremental Backup)

IMaintenance Window (Incremental Backup)

Maintenance Window (Full Backup)

ECRMS Operational Window

S is deployed and begins to process both daily and legacy PST files, it will become apparent to how much time is required for the incremental backups. If the daily 4-hour maintenan e window can be shortened, -#ti~ yv ill fillo w more time for processing of PST files.

/'. J

-... lIJlII!,e dIQW'C,\

Also, t e 12AM tart of the maintenance windows may change (e.g., shift a couple hours back or

forth) to ommodate the processing of the PST files, as more experience is gained with

the processing of the daily and legacy PST files. The ECRMS maintenance windows must also be coordinated with the IS&T backups, since they cannot overlap. The files resulting from the ECRMS data backups will be written to tape during the IS&T backups.

Also during the ECRMS Maintenance Windows, the Trend ServerProtect daily virus pattern update and the weekly full virus scan of each ECRMS server should be scheduled. This is highly recommended since the Trend ServerProtect daily virus pattern update has been known to cause

ECRMS SQL errors. \

L uJl\<h\~' ~Ov\oJ3

l~\tJ~~~: ~ \\\#

MARCH 27, ZOOS - ECRMS BACKUP AND RESTORE PUUi

GEORGE W. BUSH PRESIDENTIAL RECORD

PAGES

OAP00044390

ECRMS BACKUP PLAN

*** The Centera replication can be continuous or scheduled (e.g., to occur during the ECRMS

Maintenance Window.) ~\\~\.l\P ~ •

~It- ~~.I

Additional Notes: ~'\;.

- The naming convention fo Database backup files will be:

<DB name> Full date> bak

<DB name> Increm

<date>.bak

- The naming convention for Transaction Log backup files will be: <DB name> _tlog_ <datetime>.tm

- These .bak and .tm files will be backed up to tape as part of the standard IS&T backup procedure.

2.2 Maintenance Windows

The desired maximum duration of the daily ECRMS maintenance window is 4 hours, thus allowing 20 hours for ECRMS operation. The following diagram depicts a 4-hour maintenance window from 12AM to 4AM Tuesdays through Saturdays inclusive and a maintenance window all day Sunday beginning at 12AM. If needed, the Sunday maintenance window can continue until Monday at 4AM.

EOP may opt to perform full backups on a less frequent basis (e.g., every 2 weeks or every month) than the weekly basis depicted below. This would allow more contiguous days with a 4- hr maintenance window and 20-hr operational window.

/

MARCH 27, 2006 - ECRMS BACKUP AND RESTORE PLAN

PAGEl

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044391

ECRMS - BACKUP PLAN

2.3 Backup Considerations

Searches and exports cannot occur during the Maintenance Window. If a search or / ~/{4 + export is in progress at the beginning of a Maintenance Window, the search or export will Y'tsta.\.T{£.t ~ P not complete successfully. For lengthy searches or exports, the incremental/full backup ft

scripts should be postponed until the search/export operation completes. ~fsYt

Note that the Enterprise Vault searches can be started in Read-Only mode during a

Maintenance Window, which would allow searches and exports to occur during the

Maintenance Window. However, in order to place the Enterprise Vault services in Read-

Only mode, they must first be stopped. This stopping of the services will disrupt any

search or export in progress.

It is highly recommended that a static database backup be performed periodically (e.g., quarterly) on each ECRMS database, where SQL is stopped and the .mdb file backed up. This will capture updates such as service pack (e.g., stored procedures) and user updates. These backups provide a starting place for a restore of the databases.

Advanced backup techniques like snapshots can greatly reduce the amount oftime required for backups, thus shortening the Maintenance Windows. This may bea consideration, as the size of the databases and particularly the size of the full text indexes (expected to be 3TB after 3 years of deployment) increase.

Additional backups before and after Enterprise Vault configuration changes and upgrades are highly recommended.

flo.ust ~

For scalability and resilience each Enterprise Vault "Vault Store"~ h its own SQL database for all the metadata pertaining to the objects managed wit n that specific Vault

Store. As new Vault Stores are created nrak ~nr€l that the backup afe-configured to ~.:W- (£tv-

backup the new database. Note that FileSurf typically interacts with only one Vault 5 tklV\

Store; another Vault Store may be set up for testing. J ctt!~I{ to'rt.-\

i.e :>(cme ... (J, ~ V~ ~i\Vl..

Each Indexing Servic~tore its indexes in multiple locations. Use the Enterprise ~ \d Uf l

Vault Administrator Console to examine the properties of each Indexing Service to .

determine which folders need to be backed up. Note that in the initial deployment of

ECRMS, there is only one Indexing Service and it is currently set up to store its indexes

to only one location ~:\ECRMSIndexes).

MARCH 27, 20(}6 - ECRMS BACKUP AND RESTORE PLAN

PAGE1iJ

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044392

EeRMS - BACKUP PLAN

\~~r:,~~~

3.0 RESTORE PROCEDURE • ftfJ_II)?"'() ~ • g d'.-' f:#JN so;

_ ~U</.- Jif~~O'

The Restore Procedure for ECRMS will be a series o\:anual\tt~ aided by scripts where ~";;' iL possible. ~~

~-br~L"'--'

Of'~~ ~

1. Stop the ECRMS services.

A script will be developed to stop the FileSurf services, then delay to allow their shutdown, then stop the Enterprise Vault services.

2. Restore all ECRMS databases and restore the ECRMS full text indexes. This can be approached in one of two ways:

a) Restore the last full backup, followed by all subsequent incremental backups, for all ECRMS databases and the full text indexes. Apply the necessary transaction logs for all

the ECRMS databases. defil\L ~ <.j(( ( LLJt CARf!._

~

b) Restore the last full backup, followed by all subsequent increme al backups, for all ~ <c-

ECRM.S databases ~d the ~ll = inde~es. Rep:-ocess the PS fi s that were in process ~) at the time of the failure. This option will result tn orphane CLIP on the Centera,

which EMC can remove if desired.

3. Start the ECRMS Services.

A script will be developed to start the Enterprise Vault services and then start the FileSurf

services.

;/ Is'/4is

Additional tools are available from Symantec to aid in the restoration of data, including: il1f!f1t'Yl"lidfl

ManageIndexReplay - rebuild the full text indexes for a specific Enterprise Vault archive I I1Il~! If

from the emailsstoredontheCentera.This tool is included in the EV software release. i net) I\!.t~

During testing, the rate achieved was 8-9 emails per second. Note that beginning with I to (ll'IY"Pl.ik

EV6 SPI, the utility IndexVolumeReplay will be replacing ManageIndexReplay; this I ~~u.

new utility will be loca~~1, ,":;.}_. U>t-l::h:tv . J ~ I (tJ<tI,-)

IndexChecker __..:::, ~ r;- v-f)' uu ~vt\.O~ Q_ !£ru_1<.~

~fJPO O./0L I. .. \I _ ('1:. I :r:,~t.

EVCenteraCheckup ~ncIle Enterpnse Vault with the existing Centera clips. For 0 Nt- ~ , each dis.crep~ncy, rebuild the Enterprise Vault S~L information and index information. wJJL

To obtain this tool, a call must first be logged with Symantec support and then the t10l . OJ1?A.J..».

requested. .' _.r-trJ,i' I IlJn- .

i o»: I ~

!~~ /()~~~

MARCH 27, 2J)Of) - ECRMS BACKUP AND RESTORE PLAN

PAGE1'!

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044393

ECRMS - BACKUP PLAN

4.0 BACKUP!RESTORE TESTS

Three ECRMS Backup/Restore tests will be performed:

• Test I - simulate the corruption of an ECRMS data:

7. backup and restore of the ECRMS data residing on the ECRMS servers;

• T est II - simulate the failure of the primary Centera: 7 switch-over to the secondary Centera;

• Test III - simulate the failure of the ECRMS servers: 7 complete rebuild of the ECRMS servers.

Test I will be performed on the 'production ECRMS servers' and consist of:

• One full and two daily incremental backups of the ECRMS data (databases and full text indexes) using the master scripts, with transaction logs being backed on their regular schedule.

• After the full backup and each of the incremental backups, additional emails will be ~

processed by ECRMS. / ~... .' 1-!

• Complete Checklist 1, recording all results. <checklist to include various counts and ,.\ .£tvY'

search results> ~ {> v.I-

• Restore the full backup, followed by the 2 incremental backups. Apply the subsequent '('vUf\. ,.I

transaction logs. This restore applies to the ECRMS databases and the full text indexes. j(JJAr\.OA+r-

• Complete Checkli.st 1, recording all results and comparing to the results prior. to the .- 1

restore. ~ /J,K1~ .. ~'(~I~. )M-.L ~ ;r-4f-{~ ~~k <~. ~.

'v--"{)(t ~tJ !;(f:1'·:~~,~f·ef' 1l: (It"!: ~8»C ,. a-cr--

Test II will be performed on the 'production ECRMS servers' alliltllnvolves swi chi g from the f!Jts:i

primary Centera to the secondary Centera. The steps include:

• Complete Checklist II <checklist to include opening a number of emails and the export of some emails>

• Pull the network cable from the primary Centera. Enterprise Vault should then automatically redirect its operations to the secondary Centera.

• Complete Checklist II and compare the results to those prior to the switch-over.

OA'.V)P-.OC\O~ \$-\1: $\S ~

ackups of the 'production ECRMS servers' and restores to 3 'test' servers r the duration of this test.

• One and two daily incremental backups of the ECRMS data (databases and full text

indexes) using the master scripts.

• One full and two daily incremental backups of the 3 ECRMS production servers performed by Legato Networker.

• After the full backup and each of the incremental backups, additional emails will be processed by ECRMS.

~ • Complete Checklist 3, recording all results. <checklist to include various counts and

search results and opening of emails >

rY ~ < '

)\"J~ ..

l'IifARCH 27, 20()$ - ECRMS BACKUP AND RESTORE PLAN

PAGE 12

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044394

ECRMS - BACKUP PLAN

• To simulate the loss of all 3 ECRMS servers, take them offline. To simulate a problem with all ECRMS equipment at Dakota, also pull the network cable from the primary Centera. The remaining steps are performed on 3 other servers, made available for this test.

• Load the standard EOP server image on the 3 'test' servers.

• Restore the system files, system state and application files from the full and incremental backups performed by Legato Networker.

'. Verify that the following software has been restored. If there are any concerns, then reload this software, as required, on the 3 ECRMS servers.

o SQL Server 2000, .NET, MDAC, MS XML, MSMQ, Exchange System Manager, Outlook.

• Redirect ECRMS to the secondary Centera.

• Restore the ECRMS full backups, followed by the 2 incremental backups. This pertains to the ECRMS databases and the full text indexes.

• Reprocess the PST files that were being processed at the time of the failure.

• Complete Checklist 3, recording all results and comparing to the results prior to the restore.

All three backup/restore tests are expected to take a total of7 days, broken down as follows:

Test 1: one day Test 2: one day Test 3: five days

ApPENDIX A - CHECKLISTS FOR BACKUP!RESTORE TESTS

Checklists will be created at the beginning of each test, based on the data present in the system at that time. The completed checklists will be appended to this plan.

{leL<t ~ta~Ao.,\o\.~'~ ~(,t.-h~t ~"W\ h

(0,),0 "tt.uJ.1}.u ~(4 a.ll

ApPENDIX B - SYSTEM AND APPLICATION FILES BACKED UP fO~Si \>\;. \(t,IA.l.A.(OJt -T

~d .ws \.UU>.. (At~ I The standard IS&T backup procedure will back up all drives on the 3 ECRMS servers, with the t,'l. Who)

exceptions noted below. The backups are performed by Legato Networker. VJWl,V\ ..•

Exclusion:

All files in_'s and _s E:\MSSQL directory

Other exceptions to be noted after Legato Networker has been tested on the ECRMS

~~\ W \~~~plJ>v \)J~~GJl'~~~. ~~~ ~

servers.

MARCH 27, 2()(){) - ECRMS BACKUP AND RESTORE PLAN

PAGE 13

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044395

Sign up to vote on this title
UsefulNot useful