DP File Library

hp OpenView
storage
data protector 5.5

Advanced backup to disk

- File Library

© 2004 Hewlett-Packard Development Company, L.P.

The information contained herein is subject to change without notice

Version: A.05.50

Date: July 2004

Contact: NSS – SSD Data Management

Technical Marketing

Hewlett-Packard Co.
Network Storage Solutions

-1-
DP File Library

Table of Contents
Advanced backup to disk – File Library....................................................... 3
Contents .............................................................................................. 4
Overview – Advanced backup to disk ....................................................... 5
Positioning of backup to disk technologies.................................................. 7
Backup to disk-based devices - benefits...................................................... 8
What is a File Library ........................................................................... 10
Concept (1)........................................................................................ 11
Concept (2)........................................................................................ 12
Use case – example for “Disk Staging” .................................................... 14
New file library license: B7038AA – Example: ......................................... 15
Implementation in GUI: create a file library............................................... 16
File library – repository: define directories / number of writers .................. 17
File library – define media type ........................................................... 19
File library – summary pane ................................................................ 20
File library – repository: directory properties ........................................... 21
File library: drive / writer properties – general tab .................................. 23
File library: drive / writer properties – settings tab................................... 24
File library: file depot properties – general tab........................................ 25
File library: file depot properties – info tab............................................. 26
File library: file depot properties – objects tab ........................................ 27
File library: file depot properties – usage tab.......................................... 28
File library – media view .................................................................... 29
File library – device view.................................................................... 30
File library: new notification ................................................................ 32
Changes in CLI / mmdb and configuration .............................................. 33
Compatibility / what is supported .......................................................... 38
Limitations / performance / supportability ................................................ 39
Lab for file library: ............................................................................... 43

-2-
 DP File Library

Advanced backup to disk – File Library

-3-
DP File Library

Contents
• Overview – Advanced backup to disk - benefits
• Positioning of backup to disk technologies
• What is a file library
• Concept
• Implementation in DP GUI:
− how to create/configure a file library
• Changes in CLI
• Configuration files
• Compatibility / what is supported
• Limitations / performance / supportability

July 29, 2004 2

Contents

This is a content list of items which will be presented and discussed in this training
chapter.

-4-
 DP File Library

Overview – Advanced backup to disk

Why backup to disk?
•Benefits
− Fast random access allowing:
• Fast single file restore
• Space efficient backup
− Instant recovery
• e.g. switching for recovery instead of restoring data
from tape
− Faster restore of backup chains
− No tape streaming problems
− Ideal for RAID 5
− Easy to detect disk failures
− More protected, e.g. it’s not easy to steal a disk

August 5, 2004 3

Overview – Advanced backup to disk

Customers have requirements for increasingly larger, faster methods of backing up

and restoring data. In addition, it has become more and more important that the
time required for data backup and restore should be reduced to null so as not to
interrupt the day-to-day running of company applications.
This may be achieved using split mirror/snapshot technologies in Zero-Downtime-
Backup configurations to create replica of the data or through usage of disk-based
devices which write backup data into files residing on the disks. The ZDB concepts
are not topic of this training chapter; instead the concepts of backup to disk-based
devices will be discussed in more details.

Many applications and databases frequently make small changes to existing files or
produce many new files containing business-critical data throughout the working
day. These files need to be backed up immediately to guarantee the data in them
will not be lost. This requirement means that a fast medium which can store large
amounts of data that works without interruption is necessary for storing data.
Disk-based storage media has become increasingly cheaper in recent years. At the
same time, the storage capacity of disks has risen. This has led to the availability of
low-cost, high-performance single disks and disk arrays for storing data.
Disk backup (also known as disk-to-disk backup) is becoming ever more important.
In the past, tape storage was the favored medium for backup and restore because
of its price and effectiveness in meeting disaster recovery requirements. Today,
more and more businesses are augmenting their tape storage backup solutions with
faster disk-based backup solutions. This ensures faster data backup and recovery.

-5-
DP File Library

On the slide above you see the advantages of a backup to disk solution.
Of course there are some disadvantages compared to tape to consider as well:
− Bad price performance ratio for sequential read/write
− Remote replication to a safe place might be quite expensive
− Not virus resistant, e.g. Windows to Windows backup
− Most disks are not designed to be moved
− High power consumption
− Low MTBF (Mean Time Between Failure)
− ATA: 3 years with 80% idle time
− SCSI: 3 years, no idle time
− No hardware compression build in

-6-
 DP File Library

Positioning backup to disk technologies

ZDB / IR Advanced File Device
Backup to Disk
Replication Virtual Tape, Disk Stage D2D(2T) Virtual Tape

P
t0 t1 t2
How? mirror, snap… backup to disk backup to disk

Instant Recovery yes no no

On-line Application b. yes yes yes

Easy single file restore no yes no

Easy setup yes yes no

Incremental backup no yes yes

Cheap disk no yes yes

Operating systems HP-UX, Win, Linux, Solaris all all

Licensing per TB per TB Per file device “drive”

Use case 7* 24 DBs, appl many limited

July 29, 2004 4

Positioning of backup to disk technologies

This is an overview of different backup-to-disk technologies which helps to position

these and compare the advantages and disadvantages of each.

-7-
DP File Library

Backup to disk-based devices

• DP offers three different device types:
− file device (standalone)
− file jukebox
− file library

What is a file library?

• Group of files in specified directories where backups
are written to
• User friendly, easy configuration/modification
• Recommended for unattended backups
• Capacity only limited by available disk space
August 5, 2004 5

Backup to disk-based devices - benefits

There are many situations in which it is advantageous to use disk-based devices

when performing backups. Disk-based devices are, in fact, specific files in
specified directories, to which you can back up data instead of (or in addition to)
backing it up to tape. The following list indicates some situations in which disk-
based devices are particularly useful:

 Many applications and databases continuously generate a high number of

files or changed files, containing business-critical data. Under these
circumstances, it is necessary to continuously back up the files concerned, in
order to guarantee the capability of restoring them without data loss.
In these environments, tape devices typically have to operate in stop/start
mode, because they don’t receive a constant data stream. This may result in
the tape device limiting access to the files concerned. In addition, the lifetime
of the backup device may be greatly reduced.
In this case, backup can alternatively be performed to any disk-based device,
overcoming the limitations described. As a short-term backup solution, this is
adequate in itself. If a longer term backup solution is required, the data in the
disk-based devices can be moved periodically to tape to free up the disk
space. This process is known as disk staging.
 In environments that have fast, high-capacity disk drives and slow tape drives,
you can shrink the backup window by performing backup to disk-based
devices first and moving the data to tape later.
 Disk-based devices are useful for providing fast restore capability for recently
backed up data. For example, backup data could be kept in file devices for

-8-
 DP File Library

24 hours to enable fast, convenient restore without the need to stream data
from tape first.
 Mechanically, a disk-based device is quicker to use than a tape. When using
a file device there is no need to mount and unmount a tape. When backing
up or restoring a small amount of data, a disk-based device is quicker
because it does not need the initialization time that a tape drive requires.
With a file device there is no need to move a robotic to load or unload
media, which consumes more time in a small backup or restore. This is
especially true when restoring from an incremental backup.
 The risk of media problems such as faulty tapes and tape mounting failures
are reduced to a minimum. The availability of RAID disk configurations
provides protection of data in cases where a disk fails.
 Overhead costs are reduced because there is no need for tape handling, for
instance when performing incremental backups to disk.
 Overall, disk-based storage space is becoming increasingly cheaper than
tape-based storage.

DP offers three different device types

Data Protector has a selection of devices which are designed to do backup and
restore to and from disks. These devices are referred to as ‘disk-based devices’
because they are designed to backup data to disk as opposed to tape. The
devices vary in their functional sophistication and expected uses.

The devices are:

File Device (standalone)
The file device is the simplest disk-based device. It is a standalone device and it
has to be configured manually. It consists of a single slot to which data can be
backed up. It is not possible to change the properties of the device once it has
been created. The recommended maximum capacity of data that can be backed
up with the standalone file device is up to 2 TB, if this filesize is supported by the
operating system on which the device is running.

Jukebox
The jukebox device is a logical equivalent of a tape stacker. It contains slots whose
size is defined by the user during initial device configuration. This device is
configured manually. The jukebox properties can be altered while it is being used.
Each slot in the file jukebox device has a maximum capacity of 2 TB. The device’s
maximum capacity is equal to:
Number of slots x 2 TB

-9-
DP File Library

File Library Device

The file library device is the most sophisticated disk-based device. It consists of
multiple slots to which you can backup data. It is designed to carry out unattended
backup and restore of large amounts of data. It can be automatically configured
using a wizard in the Data Protector GUI. As with the Jukebox, the recommended
maximum storage capacity of this device is limited only by the amount of data that
can be stored or saved in a filesystem by the operating system on which the file
library device is running.

Out of the three disk-based devices, the file library device is recommended for use
as an unattended backup device.

What is a File Library

The file library is a new device type introduced with DP5.5. Basically it is a group
of files in one or more configured directories to which you back up data instead of
writing to a tape. The files contained in the file library are called “file depot”. There
is no maximum capacity for the file library device that is set by Data Protector. The
only limit on the size of the device is determined by the maximum size of a file
which can be saved in a file system on the operating system on which the device is
being run. For example, the maximum size of the file library device running on
LINUX would be the maximum size of a file you can save on this operating system.

You specify the capacity of a file device when you first configure the medium. It is
possible to re-set the sizing properties of the file library at any time during use of the
device in the Data Protector GUI.
The file library device can be located on a local hard drive, or even on a network
share, as long as Data Protector knows its path. The directory path is defined at
configuration of the file library device.
However, it is recommended to use a local disk or a disk in SAN. Disks connected
via NFS/CIFS links provide only a slow connection which also is not reliable.

- 10 -
 DP File Library

Concept (1)
• File library is a disk-based device
• Backed up data is written into files on disk in DP
tape format
• Similar concept as for File-device and File-Jukebox
• Consist of configured directories used as container
for files
• These files are called “file depot”
• By default each backup/copy session creates a
new file depot
• Each file depot has a unique file name
August 2, 2004 6

Concept (1)

A file library consists of configured directories which include files where the data is
stored into. The directories are configured at the initial configuration of the file
library device. The files inside are called “file depot” and they are created each
time a backup or copy session is made to the file library. If the amount of data
being backed up is larger than the maximum file depot size, Data Protector creates
more than a single file depot for a backup session. The backed up object will span
over two file depot’s. A file depot is equivalent to a tape media in a slot, whereas
the directories represent the repository (slots) part of a library. As a consequence,
many of the known media operations can be applied, e.g. scan, format, recycle,
export, etc. However, some operations are not available, e.g. eject.

The name of each file depot is a unique identifier which is automatically generated
by the system. It looks similar to the Data Protector media ID, but actually it is not a
media ID, instead just a unique file name. Example (including path):
“C:\data\backup\0100007f54106d9295058c50008.fd”

Since each file depot contains backed up or copied data, a corresponding DCBF
file keeps the detail catalog information for it in the IDB. Thus for each file depot a
corresponding DCBF file exists.

- 11 -
DP File Library

Concept (2)

• Size/capacity of file depots is user configurable

• One or more drives can be configured, these are
called “writers”
• By default for each file library it’s own media
pool created
• New GUI wizard allows easy/user friendly file
library creation
• Automatic and intelligent space management
• Improved and graceful “Disk Full” handling

July 31, 2004 7

Concept (2)

The size of file depots is defined when you initially create the file library device.
During this process you specify all sizing properties for the device, including the
maximum size of the file depots. The sizing properties of the file depots, although
only entered once, are globally applied to each file depot within its directory. If the
size of data to be backed up within one session is larger than the originally
specified file depot size, Data Protector automatically creates more file depots until
the allocated disk space for the file library device has been consumed.
On Windows the maximum recommended file depot/slot size is 50 GB, although
the standalone file device has been tested on Windows with file depots of up to
600 GB. On Unix the maximum recommended file depot size is 2 TB.

Analog to the file jukebox file drives can be created as well; these are called
“writers”. The naming convention for the writers is:
<file library name>_Writer<number>

For each newly created file library, by default there will be a new media pool
created with the naming convention <library name>_MediaPool. Of course the
user can change this setting to any other existing mediapool of type ‘file’.

- 12 -
 DP File Library

New GUI wizard

In order to make the creation and configuration of a file library as easy and user
friendly as possible, a new GUI wizard has been added which guides the user
through the few required steps.

Automatic and intelligent space management

Important difference to a file jukebox is that with a newly created file library NO
slots/file depots will be created. They will be automatically added into the file
library during the usage. The customer has only to take care about the disk space
needed. By default all file depots will be non-appendable. This is very useful for
the efficient disk space management. Only one session will be stored in one (or
more) file depot. As soon the protection of the session expires, the file depot can be
re-used. Only in cases where the customer has sessions with small amount of data,
backup of logical and archive logs, the media usage policy of the media pool
should be changed to appendable.

Improved disk full handling

In the past it was a known problem: during writing data into the file depot it could
happen that there was no disk space left to complete the task. This has been solved
through pre-allocation of that amount of disk space which is needed to complete
the write task, in particular to complete the write of the catalog segment.

- 13 -
DP File Library

Use case - example for “Disk Staging”

Disk Agent
Stage I Stage II
Media Agent

Disk Agent
Stream to tape

Backup

Disk Agent
Tape
NEW:
direct restore from tape

Restore:
Fast restore from disk if
data still available there
Backup 1: Store on disk
and copy to tape on
scheduled basis; expire
backup on disk as disk fills

September 3, 2004 8

Use case – example for “Disk Staging”

The concept of disk staging is based on backing up data in several stages to

improve the performance of backups and restores, reduce costs of storing the
backed up data, and increase the data availability and accessibility for restore.
The backup stages consist of backing up data to media of one type and later
moving it to media of a different type. The data is backed up to media with high
performance and accessibility, but limited capacity (for example, system disks).
These backups are usually kept accessible for restore for a period of time when a
restore is the most probable. After a certain period of time, the data is moved to
media with lower performance and accessibility, but high capacity for storage,
using the object copy functionality.

There are some use cases where a file library as part of a disk staging concept is
very useful, e.g.:
- continuous backup of transaction log files (no overhead through media
load/unload and for tape drives there is no issue with start/stop mode)
- backup of slow clients without multiplexing
- tape-less backup of branch offices
- works similar as a virtual tape library

- 14 -
 DP File Library

New File library license: B7038AA

Example:
3 x B7038AA: Advanced backup to disk extension
2 × B6951AA: Drive extension for UNIX, NAS, SAN

Disk Agent
Media Agent Media Agent

Disk Agent
Stream to tape
2.5 TB

Disk Agent Use case: store on disk and shared

copy to tape on scheduled HP MSL5060
basis; expire backup on Tape Library
disk as disk fills 2 drives, 60 slots

July 30, 2004 9

New file library license: B7038AA – Example:

This new product number B7038AA includes the license-to-use (LTU) for 1 TB of
backup disk storage. It is required once per terabyte (TB) used backup disk storage
in one DP cell and is platform independent (e.g. same license for UNIX and
Windows systems). Used backup disk storage is defined as the space occupied by
protected backups and could be distributed over multiple disk arrays and systems.
Any RAID overhead is excluded and thus does not need to be considered.
The new file library requires only the B7038AA license(s), it does not require any
drive and library LTU’s, regardless how many ‘writers/drives’ and ‘filedepots/slots’
are configured.

- 15 -
DP File Library

Create a file library in GUI

July 29, 2004 8

Implementation in GUI: create a file library

A new DP GUI wizard makes it very easy and user friendly to configure, maintain
and use the file library. Only in the case of fine-tuning, the user has to have deeper
knowledge about Data Protector. This will attract more customers to use the file
library as a backup device.

In previous DP version (DP5.1 and older) backup to disk was implemented using
the standalone file device and the file jukebox, which contains file devices. The
creation and usage of the standalone file device and the jukebox with file devices
will be kept as it is. This includes the known licensing modes as well.

The creation of a file library consists of only four steps:

1. define name, description and associated client host
2. define directories and its properties, define # of writers
3. define media type (which is file anyway)
4. check the summary and click finish to create it

On the above slide you see the device creation wizard. Its first window allows
defining the file library name and its description. As the device type you select from
the pulldown menu ‘File Library’. And you have to select the associated client
system from which the file library will be controlled and used.

- 16 -
 DP File Library

File library – repository

define directories / number of writers

July 30, 2004 9

File library – repository: define directories / number of writers

In this page specify a directory or a set of directories where you would like the file
library to reside. You can use the browser (click on browse button to open the
dialog) to select the directory to be used as container for the file depots. Multi
selection is not possible in this browsing dialog.
Please note: the directories has to be on different filesystems(!) (this is enforced) and
they must exist on the disk (DP will NOT create them).
Also important: the disk on which the file library device will reside must be visible in
the filesystem. Create the directory(ies) on either an internal or external disk, e.g.
C:\backup\filelib_dir1\
D:\backup\filelib_dir2\

If the directory is on a network share (windows), then it must be entered manually

since browsing is not possible.
It is recommended that the disk on which the file library resides should be local to
the media agent, otherwise there could be an impact on performance.

IMPORTANT:
It is critical that the directory created for the file library is not deleted from the disk.
If it is deleted, any data within the file library device will be lost.

At the time the file library is created it does NOT contain any file depots. File
depots will be created as needed when a backup is made to the device.

- 17 -
DP File Library

Setting file library properties

In the same wizard dialog you can define specific property settings for the created
directory(ies) - select directory and click on properties button. A new properties
dialog will open; there are a few options that can be changed here. These must be
set for each created directory separately, but can be changed any time during
device usage. These options will be described in detail on one of the next pages.

You can also define the number of writers to be configured for this file library. The
default value is the number of configured directories. The naming convention for the
writers (drives) is:
<FileLibraryName>_Writer<Number>.
This name as well as the other “Writer” properties can NOT be changed at the
point of creation of the file library, but any time later you can modify the writer
(driver) specific properties by selecting the writer and opening its properties
window. A writer is the equivalent to a drive! For each writer which is used to write
data during backup a separate BMA will be running. Increasing the number of
writers might improve the performance, but will also consume more system
resources, e.g. memory.

For each newly created file library there will be a new media pool created. The
naming convention for this media pool is <FileLibraryName>_MediaPool.

- 18 -
 DP File Library

File library –
define media type

July 29, 2004 10

File library – define media type

In this page please select the type of media. The only available selection is the type
“File”.

- 19 -
DP File Library

File library –
summary pane: click finish to create

August 2, 2004 13

File library – summary pane

This page is the summary pane and it shows you details of the current file library
device settings. It includes the following information:

- Directory name: the full path to where the file library device is saved.
- Total size: the total amount of space allocated to the entire file library device.
- Used: the amount of space allocated to the file library device which has already
been used for backups.
- Maximum available disk space for backup: the total amount of space available to
save data in the file library device.
- Maximum File Depot Size: the maximum size of a single file depot.

- 20 -
 DP File Library

File library – repository

directory properties

July 30, 2004 12

File library – repository: directory properties

The properties for a file library are divided into four tabs:
1. General
2. Repository
3. Settings
4. Summary

For each configured directory you can specify the sizing details in the properties
dialog window. This can be accessed in the repository tab by selecting one of the
available directories.
Any changes made will be applied to each file depot created in this directory. The
properties of any file depot created before this change will not be affected.

What are the options to specify?

 Maximum size of a file depot: This is equivalent to media size of tapes
(format size). If the amount of data backed up in one session is bigger than
this value, a new file depot will be used to continue the backup. The default
value is 50 GB.
 Minimum free disk space to create new file depot: The default value is 2
MB.
 Amount of disk space which should stay free on disk: The default value is 0
MB
 Event if the free disk space drops below (%): An event will be triggered if the
free disk space drops below this value. The default is 10%.

- 21 -
DP File Library

This table shows the default values as well as minimum and maximum values for
each setting:

Data Protector calculates these settings as shown in the following example:

Disk Properties
Total directory size = 100 GB
Used directory space = 20 GB
File Library Device Properties
Free disk space setting = 30 GB
Available Writable Space Calculation
Available space minus Free disk space 80 GB - 30 GB = 50 GB
Available writable space = 50 GB
Data Protector performs the calculation above before using a file depot. If the
amount of space is insufficient, Data Protector sends a message to notify you. This
check is made at the start of each back up session.
If this is below the minimum of free disk space required to create a new file depot a
special mount request (space request) will be issued to ask for free disk space.

- 22 -
 DP File Library

File library: drive / writer properties –

general tab

July 29, 2004 13

File library: drive / writer properties – general tab

If you open the Devices & Media context and expand the devices item, you will
see the list of configured devices.
After you expand the file library item you can see two more sub items: directories
and drives. If you expand these both as well you can see the directories structure
with the created file depots and the created drives.

How to add more writers / how to access the writers properties:

During the initial file library creation you can only specify the number of writers to
be used. Default value is the number of configured directories.
However, later at any time you can add more writers to the file library (simply click
right mouse button and select “add device”) and you can access and modify their
properties.

If you open the properties for a writer (drive) there will be two tabs available:
1. general and
2. settings

In the general tab pane you can change the default device (writer) name and its
description.

- 23 -
DP File Library

File library: drive / writer properties –

settings tab

August 1, 2004 16

File library: drive / writer properties – settings tab

In the Settings tab pane you can access and change the media pool to be used as
well as the advanced options (e.g. concurrency, block size, segment size, lock
names, CRC, etc.) and in case needed you can also disable the device (writer).
This is done the same way as for regular tape devices.

- 24 -
 DP File Library

File library: file depot properties -

general tab

July 29, 2004 15

File library: file depot properties – general tab

Similar as opening the properties for writers/drives you expand the file library
directory structure and get access to the file depot properties.
There are four tabs available:
1. General
2. Info
3. Objects
4. Usage

Under general tab you can modify the description and location. Here you can also
see the associated media label, location, and media pool.

- 25 -
DP File Library

File library: file depot properties -

info tab

July 29, 2004 16

File library: file depot properties – info tab

In the info tab pane you can view the file depot specific information such as:
- medium ID
- Quality
- Protection
- Create date/time
- Last access
- etc.

- 26 -
 DP File Library

File library: file depot properties -

objects tab

July 29, 2004 17

File library: file depot properties – objects tab

In the objects tab pane you can view which objects are included in this specific file
depot and the corresponding details:
Status: The backup status of the object on the medium.
Object Type: The type of the object on the medium.
All Media Available: Whether or not all media necessary for a restore of the object
are available.
Client System: The client system on which the backed up object resides.
Source: The location of the backed up object on the client system.
Description: The user-defined description of the backed up object.
Size (KB): The size of the backed up object in kilobytes.
Backup Type: The type of the backup (full or incremental).
Start Time: The time when the backup started.
End Time: The time when the backup finished.
Data Protection: The protection of the data on the medium.
Catalog Protection: The catalog protection for the object in the IDB.
Number of Warnings: The number of warnings issued during the session.
Number of Errors: The number of errors that occurred during the session.
Access Type: Private
Private objects can be seen and restored only by the user who created the backup,
the system administrator, or users with the See private objects user right.
Public: Public objects can be seen and restored by all users who have appropriate
restore user rights.
Session: The session ID, consisting of the date of the session and a unique number.

- 27 -
DP File Library

File library: file depot properties -

usage tab

July 29, 2004 18

File library: file depot properties – usage tab

In the Usage tab pane you see the estimated used and free space on the medium.
This pie chart displays the usage information relative to the defined maximum of this
file depot. This is visible from the device view perspective.
Similar properties exist for the specific media from the media view perspective.
However, if the media usage policy is set to non-appendable (default) the usage
there shows always 100% used (red pie chart, no free space on media) for
protected media or 100% free for unprotected media (green pie chart).
This is due to the file library concept to allocate space as needed during the
backup/copy session. There is no pre-allocation of disk space.
Simply only this amount of disk space will be allocated as needed to write the
data, not more and not less.

- 28 -
 DP File Library

File library: media view

July 29, 2004 19

File library – media view

All file depots have a corresponding media ID associated. And of course this
media is grouped in a media pool. This media pool is automatically created at the
creation time of the file library. You can view all media belonging to a media pool
and perform some operations on the media. This includes
- Export
- Change location
- Recycle
- Move to pool
- Copy
- Verify
- Import catalog
- Select media and
- Properties

Similar operations can be performed on the media pool:

- Format
- Import
- Delete
- Select media
- Properties

- 29 -
DP File Library

File library: device view

July 29, 2004 20

File library – device view

Similar as for the media view applies to the device view. By expanding devices
item, expanding file library and its directories you can view and access the listed
file depots. Similar operations as before for media can be performed here as well
on every file depot:
- Scan
- Format
- Import
- Export
- Recycle
- Import catalog
- Properties

However, there are differences which are important to know:

- Import
The import function can only be executed on file depots which have been
previously exported from this file library. You can NOT import any other file depots,
e.g. from a different system or device files created by a standalone file device or a
file jukebox. It is rather the other way around: you can import file depots from a file
library using a standalone file device or a file jukebox.

- 30 -
 DP File Library

- Export
In addition the export function provides a different result as well:
If you export a file depot from a file library, this needs first to be recycled to remove
any protection then it can be exported. Any reference will be removed from the
IDB. However, export does not mean the file depot will be totally removed from the
file library (as one would expect). Instead you still can see the file depot icon, but
the name and its details disappeared and the item is inactive (gray). Its also not
visible in the media pool.
Now you can delete these inactive file depots.

- Recycle
The recycle operation of a file library also works slightly different:
You can recycle either an individual file depot or all of the file depots in a file
library. This means that the protection will be set to none and that the disk space
which has been allocated for the recycled items, will be re-used in the next backup.
This is done by deleting the unprotected file depot(s) and creating new ones.

- Delete
To delete a file library you first need to remove the protection from all data stored in
file depot(s) of this file library. This is done by the recycle process. Then all file
depot(s) must be exported from IDB. Then you can delete the entire file library.

- 31 -
DP File Library

File library: new notification

August 5, 2004 23

File library: new notification

A new event and a notification named “FileLibraryDiskUsage” has been introduced
for the file library. This event will be triggered if the free disk space drops below a
defined percentage of total disk space. This percentage value is defined in the
properties for the directories of the file library. By default the notification is send to
the DP event log but can configured to a different send method, e.g. send email,
broadcast, SNMP, use report group or external script/command.
This check is made at the start of each back up session.

- 32 -
 DP File Library

Changes in CLI / mmdb and

configuration
with introduction of file library
• omnidownload: output has changed
• omniupload: input has changed
• new policy/device type: “Filelibrary”
• new keyword in library definition: “directory”

Changes of configuration:
• Media allocation procedure
• Media retention policy
• FileMark table extention
• ‘Disk Full’ handling

July 31, 2004 23

Changes in CLI / mmdb and configuration

Configuration data of a file library includes more data compared with a file
jukebox, therefore more information has to be stored in the IDB (mmdb part).
A new device type/policy has been introduced: “Filelibrary”
Since a file library contains a list of directories (at least one is required), a new
keyword DIRECTORY has been also added, followed by its specific configuration
parameters. These parameters are:
1. Maximum media size (in GB)
2. Minimum required space for the creation of a file depot (in MB)
3. Free disk space (in MB)
4. Value (percentage) of free disk space at which an event is triggered (in %)
5. Directories path

Example: A file library is with only one directory configured. The directory has a
size of 100 GB. Out of this 20 GB have been already used. The free disk space
has been set to 2 GB. This still leaves 78 GB to write data into this directory.

NAME "Filelib_4"
DESCRIPTION "This is a FileLibrary"
HOST testsystem.bbn.hp.com
POLICY Filelibrary
TYPE File
REPOSITORY
"/var/datastore/FL_4/dir1/0f8b594f53f7bfbe850b055038b.fd"
"/var/datastore/FL_4/dir1/0f8b594f53f7bfbe850b055038e.fd"

- 33 -
DP File Library

"/var/datastore/FL_4/dir1/0f8b594f53f7bfbfa50b05504bd.fd"
"/var/datastore/FL_4/dir1/0f8b594f53f819e2452c535021e.fd"
DIRECTORY
"25.00 100 2000 10 /var/datastore/FL_4/dir1"

Media allocation policy

The File Library will be able to create new slots/file depots on demand. At the
beginning DP will look for existing media which are unprotected (all objects,
sessions on the media are expired). These media will be deleted in the database
and in the file system. Next, DP will try to create a new file depot.
In the case that it’s not possible to create a new file depot a special mount request ,
called “space request”, will be issued.
Example:

The allocation procedure in more details:

 BMA requests new media from BSM.
 BSM asks for new media from MMD.
 MMD creates a list of all expired media in this File Library and sends this list
to the BSM.
 BSM sends this list to the BMA.
 BMA deletes the unprotected media in the file system and sends a list of
deleted media to the BSM.
 BSM sends this list to the MMD.
 After receiving the list of deleted media the MMD will delete the unprotected
media from the database.
 MMD gets all directories that are configured in the library. It has to find all
drives that are running at this time and belong to this library so it can do a
basic load balancing. MMD will sort these directories in such order that first
directory on the list is the most preferable place for creation of new file

- 34 -
 DP File Library

depot. All directories that are already used by any drive will be put at the
bottom of the list.
 MMD sends the list of directories to BSM.
 BSM forwards it to BMA.
 BMA goes through the list and checks each directory for empty space. As
soon as it finds one it sends the name of this directory back to BSM.
 BSM sends this directory name to MMD.
 MMD receives the name of directory. It creates a new file depot in this
directory.
 MMD sends the name of new file depot to BSM.
 BSM forwards file depot’s name to BMA.
 BMA formats file depot and starts writing data into it.

Media retention policy

The media retention policy for file libraries is fixed and can not be changed by the
user. It provides minimum media retention.

Example:
Media with expired data will be deleted as soon as possible. With this minimum
retention the usage of disk space is minimized which is appropriate for disk space.
The task of deleting expired media will be executed by the BMA when it requires a
new media (MSG_SLOT), which means at the start of a backup or copy session
and when a new media is requested. The deletion of expired media will be done
in the following steps:

 BMA requests a new media by sending MSG_SLOT to BSM.

 BSM sends next media request to MMD.
 MMD generates a list of expired media. MMD locks these media and sends
the list of media to BSM.
 BSM forwards this list to BMA.
 BMA deletes the file belonging to this media on the file system.
 BMA sends the list of deleted files/media name to BSM.
 BSM forwards this message to MMD.
 MMD deletes the media from MMDB.

FileMark table extention

The FileMarkTable will be extended for the file devices (e.g. standalone, file
jukebox and file library) to be able to increase the number of entries in the
FileMarkTable. This results in the capability to support file depot sizes up to the
maximum of supported file sizes on a specific OS, e.g. 1xfile depot=1xfile=up to
2TB.
With previous DP releases the FileMark table was limited to 946 FileMarks (FM’s).
The DP tape media format specification needs 2 FM’s (one DoubleFileMark) to
indicate the EOD (EndOfData) at the end of the media. Furthermore one DS and

- 35 -
DP File Library

one CS are always tightly related together. One FM will handle exact one of these
segment-offsets. This results in 470 FM-Pairs ((946 – 2(EOD) - 4(FMBuffer)) : 2) for
the DS-CS-Pairs.
The currently supported maximum DS-Size is 2GB and CS-Size is 12MB. This
results in 940GB (470 DS’s * 2GB) maximum backup data and about 6GB (470
CS’s * 12MB) catalog data.
The theoretical maximum reachable file device size is 950GB.
Note: This is a theoretical value because DS and CS will not be filled up both to
the maximum. Typically only one of them (DS or CS) reaches the maximum size.
Whereas the maximum OS-Supported file size is 2TB (e.g. supported on Windows,
HP-UX, Solaris, Linux).

Virtual FileMarks (VFM’s)

MAblkStartHdr
MANullBlock
1.

8KB 16KB DS CS … DS CS 8KB DS CS …

2. 2.
… DS CS 8KB DS CS … DS CS 8KB … DS CS
}
MAFileMarkBlock
CatalogSegment
DataSegment EndOfData

The significant changes have this pattern!

The FM-Table Extension concept is based on a change of the DP data format. A

new block type will be introduced to store the extended FM-Table. The name of the
new block type is “MAFileMarkBlock”.
The new block type will be placed after the CS. The number of FM-Blocks is not
fixed.
The size of the MAFileMarkBlock is fixed. By default the FM-Block size will be 8KB.
The MAFileMarkBlock type will be implemented only for the file device type. For
any other media types (like tapes or MO’s) this new MABlock-type will not be
implemented (not used there). To indicate the data format change the format version
will be increased.

- 36 -
 DP File Library

At the beginning of a backup the FM-Table inside the MablkHeader will be used
until this FM-Table is filled up (see point 1. in above picture). The current
implementation does stop to write the current file-depot and request another one.
The extended FM-Table feature writes a new FileMark-Block behind the current CS
and progress with the backup session. All new FM’s after the FileMark-Block will be
written to that FM-Block, until this block is filled up too. Then the next FileMark-Block
will be written (see point 2. in above picture). The FM-Table contains a new offset
value, which points to the offset of the next FM-Table.

FM1 FM 2 FM 3 … FM n …

The maximum amount of DS-CS pairs is theoretically unlimited.

If the BMA of DP 5.5 writes first time to a file media formatted with an older version
of DP ( <= 5.1) it will upgrade the media to a DP 5.5 file media, i.e. change the
minor version number to 0003 and treat this media as a DP 5.5 file media later
on.

Disk full handling

The main problem of the Disk Full handing is that the last written segment (DS-data
segment or CS-catalog segment) of a file-depot could be not valid (has not been
written successfully) in case there were not sufficient disk space left.
This has been solved based on the preallocation of the disk space which is needed
to write the missing CS (catalog segment). In this case so called SpaceDummyFiles
are created for each existing file library directory.
The pre-allocated CS-SpaceDummyFiles (CS-SDF) are a sort of dynamic space
preallocation; the CS-SDF will be checked at start of backup/copy and if required
dynamically allocated just for the catalog segment.
The “Pre-allocated CS-SDF” reserves just the amount of disk space to write a CS.
In detail: the MA creates a certain number of SDF (Space Dummy Files) each
having the same specific file size. Before the BMA starts a backup session the
SDF’s with specific number and size have to exist. If the SDF’s do not exist, the MA
will create them. If during a running backup session a DISK_FULL condition occurs,
the BMA tries the delete one of the SDF’s and to write the final CS to actual file-
depot.
A SDF should have at least the maximum size of a Catalog Segment (default
12MB). The number of SDF’s depends on the maximum concurrency (default 32).
Generally: the higher the number and the size of SDF’s the higher the security to
close the file-depot successfully in case of DISK_FULL condition.

- 37 -
DP File Library

Compatibility / what is supported

• compatibility:
− NO manual import of device files possible / supported
− DP5.0/5.1 DA compatible with DP5.5 MA (file library)
• supported platforms:
− HP-UX (including 11.23, 11.31)
− Windows 2000/XP/2003
− Sun Solaris
− Linux (RedHat, SuSe)
− AIX
− True64
− OpenVMS
− Netware

August 3, 2004 24

Compatibility / what is supported

Since the file library has been introduced with Data Protector 5.5 for the first time
there are no ‘old’ file library configurations or ‘old’ file depots where backwards
compatibility needs to be considered. In any case to configure and use a file
library, a DP5.5 MA in a DP5.5 cell is required.
It should also be possible to use ‘older’ DP disk agents to send backup data to a
DP5.5 MA to write this into a file library. However, officially it is only supported to
have all DP agents in a cell of one version (5.5).

As described on a previous slide, a manual import of device files is not possible

and not supported (either from a standalone file device or from a file jukebox).

The supported platforms for the file library are following:

− HP-UX (including 11.23, 11.31)
− Windows 2000/XP/2003
− Sun Solaris
− Linux (RedHat, SuSe)
− AIX
− True64
− OpenVMS
− Netware

- 38 -
 DP File Library

Limitations / performance / supportability

•limitations:
− Length of directory path (incl. dir name) up to max. 46 characters
− NO manual import of device files supported
− Manually formatted media will not be used
− Pre-allocated media will not be used
− Media scan only possible after first medium created
− Only ONE file library directory per mountpoint (UX) or drive (Windows)
•Performance considerations:
− Performance tuning through # of writers and concurrency setting
− Directories distributed over mountpoints (UX) or drives (windows)
− Disks shared with other disk space consuming applications
•supportability:
− Several new error messages / warnings
− File depot properties conflict
− Default block size has changed

October 26, 2004 26

Limitations / performance / supportability

Limitations

- Length of directory path (including directory name) can be up to maximum 46

characters
This is a DP internal limitation, how the file library configuration data is stored in the
mmdb. The actual limitation is 76 characters, but already 30 characters are used
for the automatically created filedepot name. This actually leaves 46 characters for
the path and directory name.
This limitation applies for the drive/slot configuration in standalone filedevice and
file jukebox as well.

- No manual import of device file possible

The file library does not allow a manual import of device files. Due to the concept
of the file library the media/file depot’s are created automatically when needed,
no manual creation is possible. Therefore, you can not import manually device files
since this would be same (or similar) as creating media/file depot’s manually.
In case you have the need to import (and read) devices files (also if created by a
file library), please use a standalone file device or a file Jukebox.

- 39 -
DP File Library

- Manually formatted media will not be used

A file library allows formatting a selected file depot or a medium used by the file
library as long as it is unprotected or expired. Of course after the format operation
the medium is still unprotected and does not have any data on it. At start of the next
backup to this file library, DP will check for any non-protected or expired media
and will delete these, including the just formatted medium. Instead for the backup
data it will create a new file depot/media and will write the backup to this.

- Pre-allocated media for a backup will not be used

By default file library uses non-appendable media usage policy. In this case you
can NOT pre-allocate media for a backup. As described in the examples above
according the file library concept, unprotected/expired media will not be re-used;
Instead the unprotected or expired media will be deleted and a new medium will
be initialized and used for the backup.
However, pre-allocation of media for appendable media usage policy works with
the file library.

Performance
Each customer configuration and disk layout is somehow different. Therefore it is not
possible to offer a setup and configuration that would provide best performance in
all these customer cases. If the customer complains about bad performance it is
essential first to look where really the performance bottleneck is and as next tries to
identify the root cause. If the result of such an analysis would be that it is because
of a poor file library configuration, you might try to improve the performance by
changing some of the configuration parameters, e.g.
- change location of the directories in order to use different, maybe faster disks
- change the concurrency of a writer or increase the number of writers
- check if there is a conflict with other applications using intensively the same
disks
- consider to use faster disks
- etc.

Supportability
The free disk space is configured per directory. DP will check for free disk space at
the creation of new file depot and at the start of a backup. The size of the new file
depot will be calculated in that way that even if device becomes full, there will be
still enough disk space available to complete the write of the catalog segment. The
problem arises if there are other applications writing to the same disk. The other
applications might use up the disk space so that there is not enough space
available on the disk after backup is done. Its recommended that DP should be the
exclusive user of the assigned disk space.

- 40 -
 DP File Library

- Only one file library directory per mountpoint (on UX) or drive (on Windows)
You can configure several directories as the repository for file libraries, but only one
directory for each mountpoint or drive partition.

File depots properties conflict

In situations where the file depots are located on a variety of disks, it is not
recommended to put file depots from two different file library devices on a single
disk. This is owing to the fact that if the properties are different, it can cause a
conflict in Data Protector. An example of a conflict situation would be if one file
library devices has properties which specify that the remaining disk space for the
file depot should be 20 MB and the other file library device specifies that the disk
where the file depot resides should have 10 MB of remaining space.

- 41 -
DP File Library

New file library specific error / warning messages:

Examples:
1.
[Warning] From: BSM@tpc002.domasti.bbn.hp.com "FS_test1" Time:
8/3/2004 12:34:35 PM
[61:7008] Invalid medium 0259880f:410a21be:05a4:002a in the
prealloc list of the device "filelib1_Writer0".
2.
[Warning] From: BSM@tpc002.domasti.bbn.hp.com "FS_test1" Time:
8/3/2004 6:06:08 PM
[61:7010] Medium 0259880f:410fb4e6:05a4:0042 in prealloc list of
the device "filelib1_Writer1" is protected and full or
have copies.
=====================================================================
3.
[Warning] From: BSM@tpc002.domasti.bbn.hp.com "" Time: 8/3/2004
12:59:30 PM
_________________________________________________________________
There is no disk space available for file library "filelib1"!
Please add some new disk space to this library.
Device Diagnostics:
File library : filelib1
Host : tpc002.bbn.hp.com
___________________________________________________________________

Default block size for all file devices (incl. file library) has changed:
Default block size has changed for standalone file devices, file jukebox and file
library from 16k to 64k. The default is now also visible in the advanced options
window for a device:

- 42 -
 DP File Library

Lab for file library:

Objective: the user should become familiar with the new file library as an important
part of the “advanced backup to disk” approach.

1. Create one (or more) file library(ies) with directories on different

mountpoints/partitions and define different specifications for each directory.
Use the default setting for number of writers. Please keep the media
allocation/usage policy on “non-appendable” (default).
Explore and verify what has been created.
Do you see any file depots created? _________________________________
Can you format a media (file depot)? ________________________________
How many writers per directory have been created? ____________________

2. Run two backup sessions using the newly created file library. Please ensure the
data protection has been set other then none. Check which file depot’s exist
and what the content is.
Conclusion: _____________________________________________________
Please note the file depot ID’s. Now remove the protection of these file depot’s
(recycle) and run a new backup session to the same file library.
Check again what file depot’s exist (compare against noted ID’s) and what their
content is.
Conclusion: _____________________________________________________

3. Export a file depot from a file library. Check in the GUI if this file depot is still
present/visible?
What options do you have to get this file depot back? __________________
Can you delete the remaining reference? ______________________________
Will the physical file be also deleted? ________________________________
How can you “clean up” the file depot list (e.g. in order not to list the inactive
references anymore)? ______________________________________________

4. Import an exported file depot into a filejukebox or a standalone file device.

For this you need to configure a device and define the appropriate slot (in case
of filejukebox) or appropriate SCSI address (for standalone file devices).

- 43 -