Server Recovery Policy Process Number: Designer: Date:

Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Server Recovery Policy

Issue V.1.2

Page 1
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Table of Contents
Restoration: An Overview............................................................................................................3
What is restore........................................................................................................................................3
A restore/recovery may be needed in following scenarios:....................................................................3
Restoration requests can also be of different types:............................................................................3
Restoration ownership:.........................................................................................................................3
Recovery using Networker............................................................................................................5
When to recover by file selection............................................................................................................5
When to recover by save set selection....................................................................................................5
Recovering data on UNIX/Windows systems...........................................................................................6
Recovering from Graphical Interface...................................................................................................6
Recovering data from the command prompt......................................................................................6
Recovering an earlier version of a file..................................................................................................7
Resolving recovered data conflicts......................................................................................................8
Viewing the volumes required for data recovery................................................................................8
Recovering by save set selection.........................................................................................................9
Recover a save set from the command prompt..................................................................................9
Recovery Logs........................................................................................................................................10
Windows Disaster Recovery.......................................................................................................11
Exchange 2007 Server Recovery using Networker Module for Exchange.............................15
UNIX Disaster Recovery using EMC Networker + HP Ignite Image level backup..............18
Oracle Recovery............................................................................................................................21

Page 2
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Restoration: An Overview

What is restore

Restoration/Data Recovery is a process to bring back to a former, original, or normal condition.

It is a process of salvaging data from damaged, failed, corrupted, or inaccessible secondary
storage media when it cannot be accessed normally.

A restore/recovery may be needed in following scenarios:

1. OS failures: Failure of operating system causing loss of access to files or data itself.
2. Hardware Failures: Disk failure and resulting loss of online data.
3. Human Factors: Human mistakes such as accidental deletion of data.
4. Application Failures: Application failure causing data corruption.
5. Security Breaches: Virus or external hackers may breach security and maliciously destroy
data, impact data integrity, and halt key operations. Physical security breaches can
destroy equipment that contains data and applications.
6. Disasters: Natural and other events such as earthquakes, lightning strikes, floods,
tornados, hurricanes, accidents, chemical spills, and power grid failures can cause not
only the loss of data but also the loss of an entire computer facility.
7. Regulatory and Business Requirements: Government regulations may require certain
data to be kept for extended timeframes.

Restoration requests can also be of different types:

1. Request to restore certain files on specific dates

2. Restoration of entire mount point / file system
3. Disaster recovery of entire server, needing to rebuild a server from scratch
4. Restore of particular user/users in Active directory in Windows Domain
5. Restoration of system state in case of Widows for AD recovery only
6. Restoration of single user’s mailbox or entire exchange information store restores.
7. Restoration of entire exchange server
8. Restoration of specific Oracle databases or entire Oracle server

Restoration ownership:

1. Responsibility of restoration of data from virtual tape library or tape lies with backup &
storage team.
2. Responsibility of backup & storage team lies only till restoring specified files or data.

Page 3
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

3. Any issues such as rebuilding Oracle database, Exchange database, rebuilding server lies
with the specified database or server team.
4. Backup team restores the Application data in form of the mount point with the same
directory structure. Application team needs to make required changes in the application
to bring it back online.

As per the current restoration policy, restore drill is performed every quarter for file system and
databases.

File system and databases are restored randomly.

Below is the log for file system restore test performed in last quarter:

uasm -r ./opt/perf/hyperhelp/xprinter/ppds/hpiip522.ps
uasm -r ./opt/perf/hyperhelp/xprinter/ppds/hpiv.pcl
uasm -r ./opt/perf/hyperhelp/xprinter/ppds/
uasm -r ./opt/perf/hyperhelp/xprinter/psstd.fonts
uasm -r ./opt/perf/hyperhelp/xprinter/rgb.txt
uasm -r ./opt/perf/hyperhelp/xprinter/xprinter.prolog
uasm -r ./opt/perf/hyperhelp/xprinter/
uasm -r ./opt/perf/hyperhelp/
uasm -r ./opt/perf/
uasm -r ./opt/
Received 68271 matching file(s) from NSR server `abkpwin1'
Recover completion time: Sat Jan  22 16:14:16 2011
You have mail in /var/mail/root

Page 4
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Recovery using Networker

Networker data restore is governed by browse and retention policies set for the client created
to backup a server in networker.

There are two main methods of recovering data:

1. Recovery by file selection (index-based recovery)

2. Recovery by save set selection

Recovery by file selection is the most flexible and easy to use method to recover data. However,
save set recovery can be preferable especially when there is a large amount of data to be
recovered or data is not available for quick access because it has passed its browse policy.

When to recover by file selection

1. The exact name of the file or directory to recover is unknown and you want to browse the file
system.
2. A directed recovery is required. A directed recovery is a recovery where data that was backed
up from one computer is recovered to another computer.
3. Directing recoveries to another client.
4. Only the files that are marked for recovery are recovered (not extra files).
5. It is important to keep the recovery operation as simple as possible. A save set recovery can
be a more complex operation.

When to recover by save set selection

1. You want to restore a large bulk of lost files, for example, in the event of a total disk failure.
In this case, you do not have to mark individual files for recovery.
2. The recovery of extra files is acceptable.
3. Quick access to the backup data is not available because the data file system browse policy
has expired. Quick access can be restored by re-creating the browsable client file index with the
scanner program. However, this can be a time-consuming process.
4. Memory on the recovery host is an issue. Recovering by save set selection requires less
memory than recovering by file selection.

Page 5
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Recovering data on UNIX/Windows systems

Recovering from Graphical Interface

1. Start the nwrecover (on UNIX) / Networker User Windows) program by typing this command:
nwrecover(UNIX) / winworkr(In Windows, can also be done by launching from programs menu)
 Use the -s servername option with nwrecover to specify a particular NetWorker server
on the network to use when recovering data. The /servers file, located in
/nsr/res/servers, contains an entry for each available server. If the -s option is not
entered and there is only one server detected, that server will be connected to
automatically. If there are no servers detected, or if there is more than one server
available, the Change Server dialog box appears, allowing you to choose the server.
 Use the -c client option to specify a particular NetWorker client on the network. If the -c
option is not used, the current client is assumed.

This will display a representation of the client file system.

2. Select the source client with the data you want to recover, and click OK (for Windows). The
local client is the default selection.
3. Select the destination client for the recovered data, and click OK (for Windows). The local
client is the default selection.
4. Mark each directory and/or file to be recovered by selecting the checkbox next to each
directory or file listed. To recover data from a scheduled backup other than the most recent
one, you must change the browse time.
5. Click Start to begin the recovery.
6. In Recover Options dialog box, select any appropriate options and click OK.

The time it takes to recover the files, depends on file size, network traffic, server load, and tape
positioning. During this time, messages appear so that you can monitor the recovery.
If the recovery is successful, a message similar to this appears:
Received 1 file(S) from NSR server server
Recover completion time: Tue Jan 21 08:33:04 2009

Note: The NetWorker User/nwrecover program does not recover any files in parallel. All files
are recovered sequentially. To recover files using parallelism, perform recoveries from the
command line.

Recovering data from the command prompt

Page 6
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

You can also perform a recovery by using the recover program. For example, to recover
/tmp/test/myfile from the server abkpwin1, type:

recover -s abkpwin1

Enter directory to browse: /tmp/test: Current working directory is

/tmp/test/> add myfile
/tmp/
1 file(s) marked for recovery
recover> recover
Recovering 1 file into its original location
Volumes needed (all on-line):
abkpwin1.001 at /dev/rmt/c2t0d0BESTnb
Received 1 file(s) from NSR server `abkpwin1'
Recover completion time: Tue Jan 21 08:33:04 2009
recover>

1. If you do not specify the -s option with the recover command, the file(s) will be recovered
from the NetWorker server defined in the /nsr/res/servers file that comes first in alphabetical
order.
2. You cannot recover Windows SYSTEM or VSS SYSTEM save sets by using the recover
command in interactive mode.
3. To avoid using the Windows version of recover.exe on Windows operating systems, do one of
the following:
. Include <NetWorker_install_path>\bin\recover.exe at the command prompt
. Ensure that <NetWorker_install_path>\bin is listed before
%SystemRoot%\System32 in the $PATH environment variable.

Recovering an earlier version of a file

You can browse previous backups to recover an earlier version of a file. This is useful if the
required data was deleted from the current file system and is not present in the most recent
backup.

Before changing the browse time to recover an earlier version of a file, verify that no files are
currently marked for recovery. If files are currently marked for recovery and the browse time is
changed, the marked files will still be selected for recovery even though they may have been
backed up after the selected browse time.

Note: If there are no entries for files backed up at the time selected in the Time field, the files

Page 7
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

that were backed up at the closest preceding time are displayed. Note the recovery in Windows
uses 12 hour format and Unix uses 24 hours format.

Resolving recovered data conflicts

By default, the NetWorker server attempts to recover data to its original location. If a name
conflict occurs, the NetWorker software can respond either automatically or on an file-by-file
basis.

The Recover Options dialog box, which appears each time a recovery is started, provides
options for resolving naming conflicts during recovery:

i. Relocate files to: Recovers selected files to the specified location.

ii. Rename recovered file: Renames the recovered files by appending each conflicting
name with .R in UNIX and prefixing each conflicting name with ~ in Windows.
iii. Discard recover file: Discards the recovered file and keeps the existing file.
iv. Overwrite existing file: Replaces the file on the file system with the recovered
version.
v. Prompt me at every file conflict (UNIX only): If selected, redisplays the Naming Conflict
dialog box for every file that has a naming conflict.
vi. Suppress further prompting (Windows only): If selected, redisplays the Naming Conflict
dialog box for every file that has a naming conflict.

You can configure the NetWorker software to resolve all name conflicts automatically before a
recovery is started by choosing the appropriate conflict resolution option in the recovery
options.

Another way to avoid name conflicts is to relocate the recovered data prior to recovery. In the
relocate field under recovery options enter the full pathname of the relocation directory.

Viewing the volumes required for data recovery

Users can view a list of the volumes needed to recover files and directories. To view a list of the
volumes required for data recovery:

1. Launch the NetWorker User/nwrecover program; select the item with the volume
requirements to be viewed.
2. Select Required Volumes from the View menu.

Page 8
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Recovering by save set selection

Save set recovery enables you to recover data from a save set rather than by browsing and
selecting files for recovery. Generally it is preferable to browse and select files for recovery.
However, save set recovery may be preferable when there is a large amount of data to recover
or when quick access to the backup data is not available because the data file system browse
policy has expired.

Backed up data that has not exceeded its retention policy is eligible to be recovered by saveset
recovery. However, if backed up data has exceeded its retention policy; it may still be
recoverable by using the scanner program if the destination where data was backed up
(tape/file type device) is not over-written.

Note: When you perform a save set recover operation, recover the last full backup first, and
then recover levels 1 to 9 and incremental backups in the chronological order that they were
backed up.

Note in Windows, only members of the Windows Administrators group have permission to
perform a save set recovery. In the case of a remote NetWorker client, members of the client
file system Windows Administrators group must have the Recover Local Data privilege to
perform the save set recover.

Only root has the permission to perform a save set recovery on UNIX. In the case of a
remote NetWorker client, root@client must have the Recover Local Data privilege to perform
the save set recover. The root@<networker_server> user automatically has this privilege.

To perform a full save set recovery of a client , in the Networker user/nwrecover window select
recover saveset option from Operation menu, select the particular version of the save set (if
there are multiple versions) and start the recovery.

Recover a save set from the command prompt

To perform a save set recovery from the command prompt, use the recover command
with this syntax:

recover -s servername -S ssid -d directory_path

This will recover the saveset in networker with specified Saveset Id. For multiple saveset’s to
restore use multiple SSID (Saveset ID) in the command.

Page 9
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Please refer to Networker Command reference guide for usage of recover command.

Recovery Logs

For troubleshooting log refer to /nsr/logs/daemon.raw file. This needs to be rendered in a

readable format using the command :

nsr_render_log /nsr/logs/daemon.raw > daemon.raw 2>&1

For recovery process using Networker user/nwrecover logs refer to networkr.raw log created in
/nsr/logs/. Use the same render command as above to render the log.

For recovery process initiated from command line, make sure you increase the logging for the
window or enable a session log to capture the output of recover command. In addition the
recovery can be further debugged by addition of –v or –D9 switch in the recovery command.

Page 10
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Windows Disaster Recovery

1. Check if the server is to be restored to same original server or a different server. In case
if it is a different server verify hardware configuration is the same as the original host
configuration. If server needs to be rebuilt by restoring system state it cannot be
restored to different hardware due to Microsoft limitations. Recovery of any VSS data to
another server is not supported by Microsoft. Server with different hardware can be
used only in case of data files recovery (i.e. files, etc and not VSS, Active directory).

2. Server team will install the OS on hardware with the exact same patches as on the
original.

During the installation, be sure to set the following configuration options as they were
prior to the disaster:

• Windows installation location

• Drive partitions
• Computer name
• Administrator account password
• Regional settings
• Date and time settings

Ensure that all drives that contained system information prior to the disaster are
accessible. The system information includes the boot partition, system partition, and the
partition that contains the databases and database logs for the system state
components.

a. Any Server Roles and Features that were installed before the disaster must be
installed before performing the disaster recovery.
b. Use the Disk Management utility (diskmgmt.msc) to create volumes (other than C:) if
they existed before. Ensure that these volumes are the same size or larger than the
original volumes.

3. If the server is part of a domain, reset the computer account in active directory before
initiating the restore.

4. Ensure that the computer name is correct and that the computer is installed into a
workgroup, not a domain. If the computer being recovered was previously a domain

Page 11
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

controller or a member of a domain, it will be restored to the correct domain when the
recovery is complete.

Please note, if the host being recovered uses DNS for hostname resolution, the DNS
server must be available or hostnames cannot be resolved. This could cause the
Networker software to be unresponsive when attempting to resolve hostnames. If the
DNS server is not available, disable DNS lookup for the host being recovered. When the
DNS server is available, re-enable DNS lookup.

5. Install the networker client/storage node software at the same location as original,
depending if the server to be restored was just a client or storage node. If the server to
be restored is Networker server then Networker server software is to be installed. For
Networker server recovery refer to the particular section. Update the “hosts” file with
Networker server name, storage node and IP address. Update the /nsr/res/servers file
with Networker server name and storage node.

6. To recover the servers always use latest full backup or a specified full backup. Restore
subsequent incremental backups to bring the system in a state prior to disaster.

7. Recovering the server is a two step process. First, recover the VSS or VSS SYSTEM save
sets. Once these save sets are restored successfully, recover the System Drive and all
other local drives.

a. Login to the client server to be recovered using local admin privileges and initiate
recovery using Recover option in Networker User. Select the server to be
restored as the source client and the same server as destination client. These
steps can be performed for servers which are within the browse policy. Recovery
can also be initiated using command line. Please refer to end of windows
recovery section for command line recovery steps.

• Windows 2003 and Windows XP Professional:

– SYSTEM STATE:
– SYSTEM FILES:
– SYSTEM DB:
Alternatively, if you are using Windows 2003 and Volume Shadow Copy Service
(VSS) is licensed and enabled, or for Windows Server 2008, Windows Vista or
later, mark these save sets:
– VSS SYSTEM BOOT:

Page 12
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

– VSS SYSTEM FILESET:

– VSS SYSTEM SERVICES:
– VSS USER DATA:
– VSS OTHER:
– VSS ASR DISK: (Windows 2003 only)
• Windows 2000:
– SYSTEM STATE:
– SYSTEM FILES:
– SYSTEM DB:
– SHAREPOINT: (for SharePoint Portal servers only)
• Windows NT 4.0:
– SYSTEM STATE:
– REPAIRDISK: (recover only if needed)
Networker software backs up user profiles in the save set that corresponds to
the disk drive on which they are located (for example, the C: save set). To
recover user profiles, recover the save set in which they were saved. User profile
folders are located in:
• %SystemDrive%\Documents and Settings for Windows 2003, Windows 2000,
and Windows XP Professional
• %SystemRoot%\profiles for Windows NT 4.0

From the Option menu, select Recover Options and In the Recover Options
dialog box, select Overwrite Existing File. If you do not select Overwrite Existing
File, the recovery process pauses before any existing files are overwritten. A
Naming Conflict dialog box appears, prompting you to indicate how the
Networker software should resolve filename conflicts. Click Start to begin the
recovery.

b. Continue to monitor the recovery process. After recovery, reboot the computer
to completely recover the system-protected files and restart services that were
automatically stopped prior to recovery.

If the host was a member of a Windows domain, it will rejoin the domain after
the reboot operation.

c. Log on to the server and again select recover option for Networker user and
select the source and destination client i.e. the server which we are recovering.
Select the system drive and all local drives. If needed you can choose to restore
select data on select drives except the system drive. Skip the Networker
installation directory from the restore selection. Choose to Overwrite existing
files from recover options.

Page 13
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

8. Please use Networker Disaster Recover Guide for detailed steps of restore. Any issues,
errors during restore will have to troubleshoot by backup & storage team involving
server/network administrators based on the issue.

9. Any applications such as exchange, SQL are to be recovered after the server recovery.
Please refer to the particular section for the same.

Page 14
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Exchange 2007 Server Recovery using Networker Module for Exchange

1. Follow the Windows Recovery steps as mentioned in Windows Recovery Section to

recover the windows server. Once the recovery of server is done proceed with exchange
recovery.

2. The recovery process would depend if the cluster, 1node or both nodes have to be
recovered.

A. For an exchange 2007 setup without cluster i.e. for standalone exchange server
recovery, insert the installation media (e.g. Exchange 2007 CD) on the server. Open
command prompt and change path to the installation media path and run the
command Setup.com /M:RecoverServer. In case you are recovering a server which
has hub transport role or this is the only exchange server then run the command as
Setup.com /M:RecoverServer /DoNotStartTransport .

Once done proceed to step to recover the Information Store.

B. In an exchange cluster setup if one node is down and exchange is running fine on
other node then you will only have to recover the failed node using windows
recovery step. This node will have to be added back to the windows cluster. Once
done install exchange 2007 with same exchange roles as original and during
installation select the Passive Clustered Mailbox role. Fail the exchange services to
this node and take them offline. Once that is done, launch the command prompt and
change to installation media path and run the following command :

Setup.com /RecoverCMS /CMSName:EXCHMBX01 /CMSIPAddress:x.x.x.x

EXCHMBX01 = Exchange Virtual cluster name

CMSName = Name of the original clustered mailbox server

After the clustered mailbox server has been recovered, the Microsoft Exchange
System Attendant service will start and then stop. You must then manually bring the
System Attendant resource online. Mount the databases back. Repeat procedure for
any additional passive nodes.

Page 15
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

In case of single node failure, database recovery from backup is not needed.

C. In case of scenario where both the nodes have to be recovered, restore both the
nodes as per windows recovery steps and setup the windows cluster. Once done
install exchange 2007 with same exchange roles as original and during installation
select the Passive Clustered Mailbox role. Once that is done, launch the command
prompt and change to installation media path and run the following command :

Setup.com /RecoverCMS /CMSName:EXCHMBX01 /CMSIPAddress:x.x.x.x

EXCHMBX01 = Exchange Virtual cluster name

CMSName = Name of the original clustered mailbox server

The CMS will be re-created using information and configuration settings stored in
AD. Repeat same steps to recover the other failed node and bring the exchange
cluster online.

Once done proceed to restore the database from backup as explained in the next
point.

(Please note that Edge Transport Server’s for exchange 2007 do not use active
directory and do not have any information stored in them. To recover just the Edge
transport server you do not need to follow exchange recovery procedure. Install the
Edge transport role on the specified server and restore just the settings from
backup). Please verify that following things are configured correctly:

• The storage groups and databases must already exist on the server, and have the
same names as the original storage groups or databases.
• The server must have the same Organization and Administrative Group name as
the original server.
• The server must have the same Exchange server versions and service packs.
• In case the physical node names and virtual node names are changed, verify that
the DNS entries are flushed.

3. Once exchange server is re-installed, launch the Exchange Management Console.

Dismount the exchange databases to be restored and in database properties verify that

Page 16
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

the option “This database can be overwritten by a restore” is checked. Once done
proceed to restore database from backup.

4. Please note that for exchange database recovery we should be able to browse the
database in Networker module for exchange i.e. saveset to be restored should be within
the browse policy (Index for saveset to be restored should be present or
restored/rebuilt to make saveset browse able for databases to be restored beyond the
browse policy and within retention policy). Saveset recovery using recover command is
not possible for database recovery.

5. Install the Networker User for Exchange module on both the nodes and launch the
same. Select the recover tab and select the Information Store to be restored. Verify that
in recover options “complete exchange recover process is selected” i.e. the transaction
logs will be replayed after restore. De-select the option to put database online after
restore. Make sure to restore the full backup followed by any subsequent incremental
or differential backups if any. Once the databases are restored try to mount the
databases. In case there are issues with mounting the databases, exchange team will
troubleshoot the same and contact Microsoft support if needed. Any issues with
networker recovery refer to NME administration guide.

Command line recovery can also be performed using the command:

nsrxchsv –c exchangevirtualsevername –s networkerservername MSEXCH:

For detailed usage refer to Networker Exchange Module Administrators guide.

Page 17
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

UNIX Disaster Recovery using EMC Networker + HP Ignite Image level

backup

1. Check if the server is to be restored to same original server or a different server. In

case if it is a different server verify hardware configuration is the same as the
original host configuration. Restoring the server from HP-Ignite image requires the
server hardware to be the same. In case of dissimilar hardware only application data
and file level data can be restored.

2. Server team will rebuild the OS from the image level backup taken using HP-Ignite.
The Images are stored on Storage with 1 month retention. In case a image is not
available the server OS will have to be re-installed manually with, kernel parameters
same as original.

During the installation, be sure to set the following configuration options as they were
prior to the disaster:

Computer properties:
• Hostname
• DNS domain name
• Superuser (ROOT) password

Please note, if the host being recovered uses DNS for hostname resolution, the DNS
server must be available or hostnames cannot be resolved. This could cause the
Networker software to be unresponsive when attempting to resolve hostnames. If the
DNS server is not available, disable DNS lookup for the host being recovered. When the
DNS server is available, re-enable DNS lookup.

3. Install the networker client/storage node software at the same location as original,
depending if the server to be restored was just a client or storage node. If the server
to be restored is Networker server then Networker server software is to be installed.
For Networker server recovery refer to the particular section. Update the “hosts” file
with Networker server name, storage node and IP address. Update the
/nsr/res/servers file with Networker server name and storage node. (Recovering
from image should put most of files in place, but verify that they are correct as the
image can be up to a month old)

Page 18
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

4. To recover the servers always use latest full backup or a specified full backup.
Restore subsequent incremental backups to bring the system in a state prior to
disaster.

a. To recover the application and user data log in to the server as root. Run the
nwrecover (/opt/networker/bin on HP-UX & /usr/bin on Linux) program. Select
the source and destination client as the servers from which to where we are
restoring. Select all the data to be recovered. Select option to Overwrite Existing
File. Please take caution to skip (de-select) the operating system files to be
skipped during restore. Refer to operating system documentation. These steps
can be performed for servers which are within the browse policy. In case GUI is
not available for recovery on UNIX servers, recovery can be initiated using
command line. Please refer to end of UNIX recovery section for command line
recovery steps.

b. Operating system boot files which may be present on the UNIX/Linux system, as
listed below are to be skipped by unmarking the directories during the restore:
– /unix
– /boot
– /etc/default/boot
– /stand/vmunix
– /stand/system
– /stand/build
– /dev
– /etc/path_to_inst
– /etc/name_to_major
– /etc/mnttab
– /etc/inittab
– /etc/grub.conf
– /etc/dfs/sharetab
– /etc/rmtab
– /etc/saf/zsmon/_pmpipe
– /etc/sysevent/piclevent_door
– /kernel
– /usr/kernel
– /proc

Page 19
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

If required, run another restore and mark only these files/directories then
choose to relocate or rename them and restore any specific files from these
directories.

c. Continue to monitor the recovery process. After recovery, reboot the computer
and verify that all services are starting correctly.

5. Any applications such as Oracle are to be recovered after the server recovery. Please
refer to the particular section for the same.

Page 20
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Oracle Recovery

Oracle database backups are taken using Oracle RMAN (Recovery Manager) API provided by
Oracle. Restore of RMAN backup is also done with the using RMAN. Oracle database restoration
is a two way process:

 Database Restore
 Database recovery

The database restore is carried out with the help of RMAN restore script provided by the
Database administrator (DBA). The value of Client, Server & media pool is provided by the
Backup Administrator.

Database restore places the Oracle control files, database files and relevant archive logs, back in
the specified location. DBA then has to perform Database recovery to bring the database up
and running.

For complete restore of an Oracle server:

1. Follow the Windows or UNIX Recovery steps as mentioned in particular section to

restore the server in case the entire server needs to be rebuilt. Verify that all
necessary dependencies for running Oracle are satisfied. Once the recovery of server
is done proceed with Oracle recovery.

2. Verify that original oracle users and groups are present and install Oracle. Re-install
the Oracle database using the same instance name/SID as original.

3. Verify that necessary Oracle environment variables are exported correctly.

4. The database should be opened in no mount mode.

5. Install Networker Client and Module for Oracle in the server to restore.

6. Oracle team will create the oracle restore/recover script to restore database.

7. After successful restore, DBA will start the database in open mode.

8. Any issues/failures during the restore please refer to /nsr/applogs/mslog.log for

RMAN errors and networker /nsr/logs/daemon.raw for troubleshooting.

Page 21
 Server Recovery Policy Process Number: Designer: Date:
Siddhartha
Patwardhan
CUSTOMER Reviewer: Date:

Process Owner: Ver #: Ver Date Approver: Date:

SAN Team 1

Please refer to the sample RMAN restoration script:

RUN {
ALLOCATE CHANNEL t1 TYPE 'SBT_TAPE' parms
'ENV=(NSR_CLIENT=OracleServer,NSR_DATA_VOLUME_POOL=POOLNAME)';
ALLOCATE CHANNEL t2 TYPE 'SBT_TAPE' parms
'ENV=(NSR_CLIENT=OracleServer,NSR_DATA_VOLUME_POOL= POOLNAME)';
ALLOCATE CHANNEL t3 TYPE 'SBT_TAPE' parms
'ENV=(NSR_CLIENT=OracleServer,NSR_DATA_VOLUME_POOL= POOLNAME)';
ALLOCATE CHANNEL t4 TYPE 'SBT_TAPE' parms
'ENV=(NSR_CLIENT=OracleServer,NSR_DATA_VOLUME_POOL= POOLNAME)';
RESTORE DATABASE;
RECOVER DATABASE;
RELEASE CHANNEL t1;
RELEASE CHANNEL t2;
RELEASE CHANNEL t3;
RELEASE CHANNEL t4;
}

Page 22