NETBACKUP BASIC TROUBLESHOOTING

Below are some basic areas to diagnose the problem.

1) Space
Check the space in the Disk Storage Units, Tapes, Disk arrays etc
2) Network Connectivity
Check the physical network connectivity via ping command
3) Services

 Verify that the NetBackup Services is being run at NetBackup Server from:
NetBackup Administration Console-->Activity Monitor-->Services

 Verify that the NetBackup Client Service is started at client from:

Start-->Settings-->Control Panel-->Administrative Tools-->Services
4) Check the Name Resolution
a. On the Master Server:
Program Files\Veritas\NetBackup\bin>bpclntcmd.exe -hn Client name

On the client:
Program Files\Veritas\NetBackup\bin>bpclntcmd.exe –pn
Program Files\Veritas\NetBackup\bin>bpclntcmd -ip <clients ip address>
b. Working with heterogeneous operating systems do not forget to make host file entries in
client as well as media master servers, this will be for name resolution.

5) Create Log folders

After creating the below log folders the logs start creating which we can help us to troubleshoot the
problem.
a. Create on client bpbkar folder under:
<install path>\netbackup\logs
bpbkar is creates a list of files which backups.
b. Create bpbrm,bptm,bpcd folders under ….Program Files\Veritas\NetBackup\logs
c. Go to NetBackup Administration Console-->Host Properties-->Master Server-->Properties--
>Logging-->under Process Specific override--> select 5 from the drop-down box for BPBRM,
BPCD
d. At client side create the folders bpcd and dbclient under
<install path>\netbackup\logs
e. Go to Backup, Archive and restore window-->File-->NetBackup Client Properties--
>Troubleshooting-->write 5 in the verbose and 9 in the database box
6) SQL Backup error – Status 2
This error will be shown up due the change in password or permission of the user added in instance,
we need to upgrade the password in the instance if it works then OK, otherwise need to delete the
that instance and create new one with the same configuration it will OK.
7) Oracle Backup error – Status 6
This error will come when there is no space available on the server, you need to spare some space
for backup activity. In our case in FFL servers there is a folder named as backups under oracle
directory where daily backup of the server is stored by ORACLE itself not by NetBackup. This was
configured by vendor to be on safe side, on the other hand we are taking the backup of the same
oracle instance through NetBackup.
Because of daily backup initiated by Oracle the space on the drive decreased and the backup thru
NetBackup shows an error with status code 6.
All we need to do is the following;
a. Login to subject server and locate backups folder which should be at following location
i. /ora01/app/oracle/backups
Delete old backup files which are of no use and then restart the backup from NetBackup.

8) Vnet Proxy Encountered an error.

Cause
At version 8.0, non-master-server NetBackup hosts automatically and silently request host
certificates from the master server.

When the host was new, the master server accepted the request and issued a certificate to the
host. Later, something happened to the host that required it to be re-installed. Whatever the
event, it likely included the host losing the original certificate.

When the host was re-installed, it requested a host certificate from the master server, as usual.
However, this time the master server had records indicating that it had already issued a
certificate to the requesting host, and so it denied the request. A certificate request requires a
reissue token in this case, but the automatic request that the host sends does not include reissue
tokens (or any other kind of authentication token).

Although the re-installed host had no certificate, backups and restores continued to work
normally because they do not require host certificates to function. The lack of a certificate went
unnoticed until the host was upgraded. NetBackup 8.1 requires host ID certificates to
communicate with non-back-level hosts.
Solution

Generate a reissue token on the master server for the host in question. Please refer to the
Veritas NetBackup Security and Encryption Guide for details on how to generate reissue tokens.

On the client or media server, run the nbcertcmd command below and enter the reissue token
when prompted. The reissue token will not be echoed to the terminal when you type it.
nbcertcmd -getCertificate -host client.example.com -server master.example.com -token

Windows: <install_path>\netbackup\bin\nbcertcmd
Unix: <install_path>/netbackup/bin/nbcertcmd

Resulting output:
 Authorization Token: [Type or paste in the uppercase, 16-character token] Host certificate and
certificate revocation list received successfully from server master.example.com.
For details and alternative methods of running nbcertcmd, please refer to How to manually obtain
a host ID certificate and follow the instructions to request a host certificate using a reissue token.

Note : If you receive error :

EXIT STATUS 5989: Reissue token is mandatory as a certificate is already issued to this host.
Revoke the existing certificate if it is active and map this host name to the associated host ID.

9) Lotus Notes Backup error – Status 103

NetBackup always shows 103 error code whenever the lotus backup fails, we have got this error two
times so far;
a. This error will be shown up after fresh installation of Domino Server over Windows Server
2016.
Cause
During the installation of any software it writes some files in Windows registry but due to shift
from windows 2012 to windows 2016 there is some architecture change due to which it is unable
to write or update registry files. There was no file was present under following registry location;
HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\

&

HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\1

Solution
We need to add files with specific binary values and description in registry as shown in the
picture below;
b. This error will be shown up after upgrading the Domino Server from 10 to 11.
Cause
Domino is acquired by HCL in year 2020 and during the upgrade from domino 10 to domino 11 it
changes the architecture of registry files, it created HCL directory then domino then 1 but
previously it was Lotus then domino then 1.
After troubleshooting we got to know that the Lotus folder present in the registry is empty and the
same hierarchy is built up under HCL folder, there is an architecture change due to which it is
unable to write under lotus registry folder, there was no file was present under following registry
location;
HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\

&

HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\1

Solution
We need to export HCL registry files edit that file accordingly as given below “Replace HCL with
Lotus at specific locations” and import them under Lotus folder present in registry as shown in
the picture below;
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Lotus]

[HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\Domino]
"DataPath"="D:\\IBM\\Domino\\data"
"Description"="HCL Domino"
"Name"="nserver.exe"
"Partitions"="1"
"SetupType"="Enterprise"
"Version"=dword:0000000b
"Path"="D:\\IBM\\Domino"

[HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\Domino\1]
"Description"="HCL Domino"
"Name"="nserver.exe"
"Path"="D:\\IBM\\Domino"
"Version"=dword:0000000b
"SelectedFeatures"="Program Files,DataFiles,Domino Enterprise Connection
Services,License,iNotes,Domino Directory Sync Services,Domino As A Windows
Service,Performance Monitoring,OS Integration,Resource Modeling Engine,Billing
Support,Clustering Support,Optional Network Drivers,Symbols Files,Java
Support,Required Templates,Administration Templates,Optional Templates,Certificate
Management,Readme Files,Dojo,Xpages,Web Services Data Files,Enterprise Server
Files,Help"
 "DataPath"="D:\\IBM\\Domino\\data"

10) Tape Erase / Tape Backup – Status 98/811

Run the following command to clear host cache on the media server on Suse Linux

# /usr/openv/netbackup/bin/bpclntcmd -clear_host_cache

Run the following command to clear host cache on the media server on Windows

/Program Files/Veritas/Netbackup/bin> bpclntcmd.exe -clear_host_cache

11) Backup failing – Socket 25 Error

Run the following command to redeploy certificate on the client/media server on Suse Linux

# /usr/openv/netbackup/bin/nbcertcmd -getcacertificate

# /usr/openv/netbackup/bin/nbcertcmd -getcertificate -force

# /usr/openv/netbackup/bin/nbcertcmd -ping

Run the following command to redeploy certificate on the client/media server on Windows
/Program Files/Veritas/Netbackup/bin> nbcertcmd.exe -getcacertificate

/Program Files/Veritas/Netbackup/bin> nbcertcmd.exe -getcertificate -force

/Program Files/Veritas/Netbackup/bin> nbcertcmd.exe -ping

12) ISSUE: Unable to expire Images from database after catalog restore incident due to which
media ID was changed from @aaaab to @aaaad.

CAUSE: Image cleanup job was partially completing

DIAGNOSTIC STEPS:

1) Checked Image cleanup job was partially failing.

2) It was trying to reach media ID @aaaab which was no longer exists.

3) Run "nbdevquery -listdv -stype server_type -dp disk_pool_name -U" for each disk pool to find out
Media ID.
dp_disk_psnbuapp--> @aaaad
dp_adv_psnbuapp--> @aaaac
PSFMLNBU_DEDUP--> @aaaak
fml_dedup_pool--> @aaaag
FMLMNU_DEDUP--> @aaaai

RESOLUTION/ WORKAROUND:

1) Run "nbdelete -list" under the path <install_path>\NetBackup\bin\admincmd\ on master

2) Then run nbdelete -purge_deletion_list -media_id @aaaab -force on master

3) After that triggered image cleanup job using the command line-

\netbackup\bin\admincmd\bpimage -cleanup -allclients on master

Cleanup job completed successfully.

13) Barcode reading issue

Command to update barcode:

Installation Path\Volmgr\bin\> vmchange -barcode <barcode> -m <media_id>

14) OpsCenter only exports 4000 lines

Cause

By default, OpsCenter is set to only export 4000 lines. This is due to the performance issues that can be
seen at the OS level when set higher.
Solution

There are 2 files that need to be modified for the default of 4000 lines to be changed. This solution
applies to all reports (scheduled and emailed or manually generated).

1. Make a copy of the below file for safety and then edit it adding the following <context-param> tag
between the </description> and </web-app> tags:

Linux: /opt/SYMCOpsCenterGui/webserver/webapps/ROOT/WEB-INF/web.xml

Windows: <install_path>\OpsCenter\gui\webserver\webapps\ROOT\WEB-INF\web.xml

</description>
<context-param>
<param-name>maxTabularRows</param-name>
<param-value>8000</param-value>
</context-param>
</web-app>

** In this example it will increase the maximum size of the report to 8,000 lines.

2. Also, browse to the below directory and open the report.conf file in a text editor and edit/add the
following parameter:

Linux: <INSTALL_PATH>/SYMCOpsCenterServer/config/report.conf

Windows: <INSTALL_PATH>\Symantec\OpsCenter\server\config\report.conf

report.schedule.max.tabular.rows=4000

* Search the file for the parameter. If it exists, edit it. If it doesn't exist, add it to the very bottom of
the file on it's own line. The example shows the default of 4000 lines. Modify this value to what you
need it to be.

Restart OPSCenter Services

1. To stop the OpsCenter services run the opsadmin command with the stop switch:

INSTALL_PATH\OpsCenter\server\bin\opsadmin.bat stop

2. To start the OpsCenter services run the opsadmin command with the start switch:

INSTALL_PATH\OpsCenter\server\bin\opsadmin.bat start

15) VM Backups start Failing with Status Code 6

DIAGNOSTIC STEPS/CAUSE:

VXMS logs 19:00:53.559 : loadBitmap:VixGuest.cpp:2443 : Clearing bitmap for unchanged region

between 1199863496704 and 694250635264 (length: 18446743568096690176) 19:00:53.559 :
clearRegion:RvpBitmapX64.cpp:199 : Exception: VERIFY_VXMS_BOOL19:00:53.559 :
 log:Error.cpp:249 : Error: 0x00000003 occured in file
D:/NB/10.1/src/vxms/plugin/common/map_v2/rvp/RvpBitmapX64.cpp, at line 19919:00:53.559 :
log:Error.cpp:249 : Error: 0x00000003 occured in file
D:/NB/10.1/src/vxms/plugin/common/map_v2/rvp/RvpBitmapX64.cpp, at line 19919:00:53.559 :
vixMapObjCtl:VixCoordinator.cpp:974 : Returning: 319:00:53.559 :
hv_vmware_map_objctl:libvix.cpp:1171 : Returning: 319:00:53.559 :
vixMapClose:VixCoordinator.cpp:303 : case VXMS_OBJECT_TYPE_GUEST19:00:53.559 :
closeGuest:VixManager.cpp:63 : Obj: C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\
bpfis.fim.FFBL-HO-MAIL_1663941602.1.0.NBU_DATA.xml19:00:53.559 :
closeGuest:VixManager.cpp:95 : Guest has no open file systems hence deleting it19:00:53.559 :
logDelay:VixVxms.cpp:149 : ID:2296635749736 Count:1, Interval(sec):2 Delay(micro):0 Size:512
KB/sec:019:00:53.559 : vdClose:VixInterface.cpp:359 : Calling Close():
00000216BA1B7AD019:00:53.560 : g_vixInterfaceLogger:libvix.cpp:1803 : [VFM_ESINFO]
VixDiskLib: VixDiskLib_GetTransportMode: Retrieve transport mode.19:00:53.560 :
g_vixInterfaceLogger:libvix.cpp:1803 : [VFM_ESINFO] VixDiskLib: VixDiskLib_Close: Close disk.
Etrack - 4088660 https://www.veritas.com/content/support/en_US/article.100054040.html

RESOLUTION SUMMARY:
Installed the EEB eebinstaller_4088660_1_AMD64 on backup host / media server.

16) Expiry of Backup Image from Tape

Use following commad
D:\Veritas\Netbackup\bin\admincmd\bpexpdate.exe -m media_id -d date | 0 | infinity
Specifies the expiration date and time. date can be any one of the following:

 mm/dd/yy hh:mm:ss
 0 - the backup or media expires immediately
 infinity - the backup never expires
For immediate expiry bpexpdate.exe -m media_id –d 0
17) Change Expiry data of Backup ID through Master Server

e.g. to recalculate the expiration date of a backup image based on a new retention level:

bpexpdate -recalculate -ret <retention level> -backupid <backup_id>

or to set a specific expiration date of a backup image:

bpexpdate -d <mm/dd/yy hh:mm:ss> -backupid <backup_id>

Ret Level Period Days

--------- ----------- ----------
0 1 week 7
1 2 weeks 14
2 3 weeks 21
3 1 month 31
4 2 months 62
5 3 months 93
6 6 months 186
7 9 months 279
8 1 year 365
9 infinity

18) Change Expiry data of all images present of Media Server

bpexpdate -servername <media server name> -d mm/dd/yyyy

19) NetBackup backups intermittently fail with status 83 when writing to a Media Server Deduplication
storage unit hosted on…. DISK STORAGE SERVER is DOWN
When the Windows firewall is disabled, the Teredo Tunneling Interface will come online and offline
intermittently. When the bptm plugin attempts to connect to the storage server services running on
the local host, the operating system may select the tunneling interface for the connection if it is
online. If the Teredo interface goes offline, it results in the errors noted above.

Solution

This is a problem with Windows 2012 R2 and onwards and is being addressed by Microsoft.

As a workaround, disable the Teredo interface on the media server. It is unstable in this
configuration and thus not being used.

1. Run netsh interface teredo set state disable.

2. Carefully modify the registry so the operating system prefers to use the IPv4 addresses over
the IPv6 addresses.
Note: Incorrect use of the Windows Registry Editor can have disastrous results on a server
operating system. Great care should be taken when making changes to a Windows
Registry. Registry modifications should only be carried out by persons experienced in the
use of the Registry Editor application. Ensure you have a backup of the registry before
proceeding.

In the Registry Editor, locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\.

Double-click DisabledComponents to modify the DisabledComponents entry.

If the DisabledComponents entry is unavailable, you must create it;

Edit menu > New > DWORD (32-bit) Value
Type 'DisabledComponents'
Enter

3. Reboot the host.

20) Clear orphan images from Catalog and free up space;

1) Windows: <install_path>\veritas\pdde
2) Process the transaction queue twice. See Process the transaction queue.
crcontrol --processqueue
3) Monitor queue processing (busy:no means its complete)
 crcontrol --processqueueinfo
4) Keep running queue processing until the queue reaches 0.
crcontrol --queueinfo
5) Once the queue is 0 and the queue processing says busy:no then start compaction
crcontrol --compactstart 0 0
6) Check that compaction is still running (busy:no means its complete)
crcontrol --compactstate
7) Check for Free space
crcontrol --dsstat