Symantec NetBackup OpenStorage Logs

and Troubleshooting Short Course

Symantec NetBackup OpenStorage 1

Sample Agenda

1 Introduction

2 NetBackup Overview

3 Processes and logs

4 Legacy logs vs. Unified logs

5 Parsing (following the breadcrumbs)

Symantec NetBackup OpenStorage 2

Sample Agenda

1 Functional Log Gathering

2 Basics, from the bottom up

3 Training from Symantec

4 Materials for this preso

5 ?&

Symantec NetBackup OpenStorage 3

 NetBackup from 5000 feet…

Symantec NetBackup OpenStorage 4

NetBackup Terminology
• Master Server
– The ‘brains’ of NetBackup
– Maintains the ‘catalog’ (the list of backup images), schedules jobs,
maintains information about all devices and systems
• Media Server
– The data movers to/from devices
– Responsible for performing the backups, duplications, restores. These are
the servers where the backup devices are ‘attached’.
• Client
– The machine being backed up, e.g., file server, application server.

Symantec NetBackup OpenStorage 5

NetBackup Terminology
• Catalog
– The list of all backups, information, retention period, etc.
• Backup Image
– Warning: NetBackup Image != OpenStorage Image
– NBU Backup images are composed of 1 to N image fragments.
– Each NBU image fragment == OST image.
• Backup Policy
– Specifies what to back up on the client (e.g., files, database), when to
perform the backup (generally multiple schedules, e.g., weekly full, daily
incremental backup), where to send the backups (e.g., disk, tape) and
how long to retain the backup image before expiring/deleting it.

Symantec NetBackup OpenStorage 6

NetBackup Terminology
• Disk Volume
– NetBackup’s name for an LSU
• Disk Pool
– A collection of Disk Volumes (LSUs)
• Storage Unit
– A means of specifying which media servers move the data to what device
(disk pool, tape library) and to what extent to throttle job activity.
– STU is the acronym, DSU acronym for Disk Storage Unit.
• Storage Lifecycle Policies
– Highly popular new way (introduced in NBU 6.5) to define a plan for how
many backup copies to make, destination and retention.
– E.g., back up to disk, make 1 copy to tape, another copy to off-site disk.

Symantec NetBackup OpenStorage 7

Before we get started…..
• I’m assuming a few things:
– You’ve got logging cranked up in verbosity
• Do this via the GUI under Server properties OR
• Vi the bp.conf file (add VERBOSE=5), for legacy logs
• Use the vxlogcfg command to up the settings for some or all unified logs
– You know how to run NBU to do backups, restores, and opt_dups
– You have all of your NBU services up and running (EMM is kind of a big
deal)
– You like to read lots of logs and take great joy in reading process flow
diagrams
– You have sufficiently caffeinated/B12’d/sugared yourself up
– You are not allergic to some perl and NetBackup command lines

Symantec NetBackup OpenStorage 8

Crank it up…
• Legacy Logs
– Edit bp.conf
– Add “VERBOSE=5” to the file
OR
– You can also simply go to your Master Server GUI and up this in the
Properties settings for the Master, it will propagate to the Media servers
• Unified Logs
– The vxlogcfg command:
• vxlogcfg -a -p <ProductID> -o <OID> -s DebugLevel=6
• vxlogcfg -a -p <ProductID> -o <OID> -s DiagnosticLevel=6
– ProductID is 51216 for NetBackup
– OID (Originator ID) – there is an exhaustive list of these in the Troubleshooting Guide (can be “ALL”)
– Run both commands for all OIDs you need to up log levels

Symantec NetBackup OpenStorage 9

 Processes and Logs (Stuff we do and
things we leave behind)

Symantec NetBackup OpenStorage 10

 These diagrams are available in our Troubleshooting
NetBackup Backup Processes Guide

Symantec NetBackup OpenStorage 11

That’s a nice eyechart, what’s it mean?
• Backup
– Three things to think about:
• Bpbrm
• Bpbkar
• Bptm
– There is a lot going on, but these are the main processes

Symantec NetBackup OpenStorage 12

 These diagrams are available in our Troubleshooting
NetBackup Restore Processes Guide

Symantec NetBackup OpenStorage 13

That’s a nice drawing of a spiderweb, what’s it mean?
• Restore
– Here, the focus will move to the bprestore, bprm, bpdm, and tar
processes
– Again, the logs will have much more to say about this…

Symantec NetBackup OpenStorage 14

NetBackup Forestry
• NetBackup Logs
– Named and produced per process
– Two types:
• Legacy Logging (older style)
• Unified Logging (newer style)
– Different processes are logged to each type of log, you will need some
information to follow your job around
• Time Window
• Process ID
• JobID from NetBackup
• Image Name from NetBackup (this is a personal favorite, not necessarily needed)
• Write that down…

Symantec NetBackup OpenStorage 15

Logs and where to find them (hint: near saws)
• Legacy logs
– Ascii text
– Placed in the (/usr/openv/netbackup/logs or <INSTALLDIR>\netbackup\
logs ) directory
– Subdirectories named for the process that logs to them
– Each log is named for its timestamp within the process directory
– Sifting through these can be tedious
• Unified logs
– Unix: /usr/openv/logs Windows: <INSTALLDIR>\netbackup\logs
– Comma separated values lists
– Viewable as text, but must be viewed using “vxlogview” command for
detail

Symantec NetBackup OpenStorage 16

Viewing logs
• Legacy logs
– A good text editor is your friend
• Vi/vim/elvis/pico/emacs/etc… on unix
• I like TextEdit on Windows
– These are basic ascii logs, arm yourself
• Process ID
• JobID in NBU
• Time frame
• NetBackup Image name
– Cd into your process directory and open the log that fits your timeframe
– Start reading…..

Symantec NetBackup OpenStorage 17

Viewing logs
• Unified logs
– Vxlogview is powerful, use it wisely
• There are a lot of options, but you need only a few to be effective
– These are not necessarily readable without the vxlogview command
– Many of the processes that are prefixed with “nb” log to these logs
– These were meant to make logging easier and to have less jumping
around between logging directories for the admin/support

Symantec NetBackup OpenStorage 18

Parsing logs
• Legacy logs
– Cd into the directory of your favorite NBU process
– You will want to search for “<16>” and “<32>” in the logs
– A timeframe is a great help here
– You can also use grep/sed/awk/perl/vbs to search the logs for
JobID/PID/imagename/time
– Scripts for tearing out the 16/32 messages are nice for doing a fast “rip
and glance” approach to narrow your search

Symantec NetBackup OpenStorage 19

QaD script to search the Legacy logs for 16/32
#!/bin/bash
ls /usr/openv/netbackup/logs | while read x
do
echo "$x:"
echo ""
cd $x
for FILE in log*
do
echo "${FILE}:"
perl -ane 'print if /\<16\>|\<32\>/;' $FILE
done
cd /usr/openv/netbackup/logs
echo ""
echo ""
done

Symantec NetBackup OpenStorage 20

Parsing logs (s’more)
• Unified logs
– Cd into the unified logs directory (/usr/openv/logs on unix)
– As of 6.5.6 and in 7.0, all processes that log here have directories
– Cd into your favorite process directory and have a look around using:
• “vxlogview –G . –d all > out”
– This will generate a LOT of logs, you can minimize it by clearing the old files you don’t want in your
timeframe out of the directory
• Also, “vxlogview –G . –d all –hoursago <#hours>”
– If you can narrow the window to “between then and now”, this is helpful
• Also try the –b and –e arguments for begin and end times for processing
– Redirect these commands to files for parsing more like legacy logs
– The vxlogview man page is very helpful, even if it is a bit wordy….

Symantec NetBackup OpenStorage 21

Time to cut some trees
• In the past, tar, gzip and Windows zip were all good to have on
hand (they still are…)
– Tar and gzip or simply archive and zip together all the log directories for
all logs and send them along to Symantec
• “Why do you need ALL of them?” - The core issue may not be the plugin or anything to do
with OST
• “Where can I put them?” - We do have a Support ftp site, that you will be given info to
login to and add your logs to your incident directory when needed
• “Can’t I just send you snippets?” -Nope, sorry. I know it’s easier, but we have to see NBU
running, from start to end, all the daemons, all the bits and pieces
• Nowadays, we have a utility, nbcplogs
– /usr/openv/netbackup/bin/support/nbcplogs
– And…. NBSU!!!

Symantec NetBackup OpenStorage 22

Basics from the bottom up
• Check to be sure NBU is running
– Sometimes it’s not…
• Check EMM to be sure IT is running
– This can cause weird issues with socket connect failures
• Check your filesystems for space (EMM needs scratch space)
• Ping all of the hosts in the config being used (master to media,
and back, media to storage and back)
– Network misconfigs or random firewall rule changes cause havoc
• Use bpstsinfo –pi
– If the plugin won’t load, it won’t backup
• Use AdvanceDisk as a baseline check of NBU vs. Plugin issue
Symantec NetBackup OpenStorage 23
Coordination between Support Organizations
• TSANet
– www.tsanet.org
– All STEP/OST Partners have an account on TSANet
– This is basic info at this point, but both sides can add detail to their
contact information
– This is a first step at handing off a case
– All STEP/OST Partners also have customer numbers and can open support
cases with Symantec on behalf of shared customers

Symantec NetBackup OpenStorage 24

Thank you!

Copyright © 2010 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in
the U.S. and other countries. Other names may be trademarks of their respective owners.

This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied,
are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

25