Troubleshooting Nokia Archive Cloud

No further reproduction or networking is permitted. Distributed by Nokia.
Copyrighted material licensed to khan.tanveer.ext@nokia.com on 12-06-2022.

Nokia Archive Cloud Operating
Documentation
Troubleshooting Nokia Archive

Cloud
Release ARC22
DN09258918
Issue 1-1
Approved on 19th May, 2022
© 2022 Nokia. Nokia Condential Information
Use subject to agreed restrictions on disclosure and use.

Troubleshooting Nokia Archive Cloud

Nokia is committed to diversity and inclusion. We are continuously reviewing our customer
documentation and consulting with standards bodies to ensure that terminology is inclusive
and aligned with the industry. Our future customer documentation will be updated
accordingly.
This document includes Nokia proprietary and condential information, which may not be
distributed or disclosed to any third parties without the prior written consent of Nokia. This
document is intended for use by Nokia’s customers (“You”/”Your”) in connection with a
product purchased or licensed from any company within Nokia Group of Companies. Use this
document as agreed. You agree to notify Nokia of any errors you may nd in this document;
however, should you elect to use this document for any purpose(s) for which it is not
intended, You understand and warrant that any determinations You may make or actions
You may take will be based upon Your independent judgment and analysis of the content of
this document.
Nokia reserves the right to make changes to this document without notice. At all times, the
controlling version is the one available on Nokia’s site.
No part of this document may be modied.
NO WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO
ANY WARRANTY OF AVAILABILITY, ACCURACY, RELIABILITY, TITLE, NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, IS MADE IN RELATION TO THE
CONTENT OF THIS DOCUMENT. IN NO EVENT WILL NOKIA BE LIABLE FOR ANY DAMAGES,
INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR
CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT,
REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA THAT MAY ARISE
FROM THE USE OF THIS DOCUMENT OR THE INFORMATION IN IT, EVEN IN THE CASE OF
ERRORS IN OR OMISSIONS FROM THIS DOCUMENT OR ITS CONTENT.
Copyright and trademark: Nokia is a registered trademark of Nokia Corporation. Other

product names mentioned in this document may be trademarks of their respective owners.
© 2022 Nokia.
2 © 2022 Nokia. Nokia Confidential Information Issue 1-1
<

Table of Contents
1 Client integration is failed ............................................................................................................ 5

1.1 Unable to contact Client ................................................................................................... 5
1.1.1 Diagnosing the problem ........................................................................................ 5
1.1.2 Resolving the problem ........................................................................................... 6
1.1.2.1 No connection between Backup Engine Node and Client ..................... 6
1.1.2.2 Ports are not open ...................................................................................... 6
1.1.2.3 Wrong Client name ...................................................................................... 6
2 Client integration known warnings ............................................................................................. 7
3 Client deleting is failed ................................................................................................................. 8

3.1 Activities are in progress .................................................................................................. 8
3.1.1 Diagnosing the problem ........................................................................................ 8
3.1.2 Resolving the problem ........................................................................................... 9
4 Backup operation is failed ......................................................................................................... 10

4.1 Scripts related causes .................................................................................................... 10
4.1.1 Scripts are not in the correct path .................................................................... 10
4.1.1.1 Diagnosing the problem .......................................................................... 10
4.1.1.2 Resolving the problem ............................................................................. 11
4.1.2 Shell interpreter in scripts is missing ................................................................ 12
4.1.3 Scripts have no executable rights ..................................................................... 13
4.2 Connectivity related causes .......................................................................................... 14
4.2.1 No connection to Data Domain ......................................................................... 14
4.2.2 Authentication error occurs in connection to Data Domain ......................... 15
4.2.3 Backups fail with Error Code 10008 .................................................................. 16
4.2.4 Data Domain GUI connection problems ........................................................... 19
4.3 Client upgrade related causes ....................................................................................... 20
4.3.1 Diagnosing the problem ...................................................................................... 20
4.3.2 Resolving the problem ........................................................................................ 21
4.4 Oracle database backup is failed .................................................................................. 21
4.4.1 Timeout on wait for sub-process ...................................................................... 22
Issue 1-1. © 2022 Nokia. Nokia Confidential Information 3


4.4.2 First backup of a big database is failed ............................................................ 22
4.5 VMware backup is failed ................................................................................................. 23
4.5.1 Datastore space limit is exceeded .................................................................... 23
4.5.2 VMware backup failed with error No proxy ...................................................... 24
4.5.2.1 Missing VM proxy (No proxy) ................................................................... 24
4.5.2.1.1 Diagnosing the problem ............................................................... 24
4.5.2.1.2 Resolving the problem .................................................................. 24
4.5.2.2 Missing VM proxy (No proxy) ................................................................... 24
4.5.2.2.1 Diagnosing the problem ............................................................... 24
4.5.2.2.2 Resolving the problem .................................................................. 25
4.5.3 VMware image backups fails with Error 10056 ................................................ 25
4.5.4 VM snapshot backup fails due to incorrect pre-evaluation of available space
.................................................................................................................................... 30
5 Alarms are raised in OMAgent .................................................................................................. 32

5.1 Alarm 2004750: Datadomain get checkpoint list operation failed is raised .......... 32
5.1.1 Password for ddboost user expired .................................................................. 32
6 Performance Managemenet data is delayed in NetAct ........................................................ 37

6.1 Performance Management data collection schedule interval is too short ............. 37
6.1.1 Diagnosing the problem ...................................................................................... 37
6.1.2 Resolving the problem ........................................................................................ 37
<

1. Client integration is failed
This section describes how to troubleshoot when Client integration is failed.
Symptoms
It is impossible to perform Client integration.
Information that operation is failed appears.
Cause
Unable to contact Client
1.1 Unable to contact Client

Client integration is failed because Client is not able to contact with port.
1.1.1 Diagnosing the problem

Diagnosing the problem when Client is not able to contact with port.
Purpose
During the Client integration, the following log appears:
2020-11-18 11:47:32,914 INFO Preparing client integration
2020-11-18 11:47:33,144 ERROR Backup Engine server is not able to resolve client
hostname: client_hostname
2020-11-18 11:47:33,145 ERROR Client: /ARC/client_hostname has not been
integrated
Possible causes and dedicated solutions:

No connection between Backup Engine Node and Client
Ports are not open
Wrong Client name


1.1.2 Resolving the problem
Resolving the problem when Client is not able to contact with port.
1.1.2.1 No connection between Backup Engine Node and

Client
Resolving the problem when Client is not able to contact with port because connection
between Backup Engine Node and Client is unavailable.
▪ Establish the connection between Backup Engine Node and Client.
1 Perform the ping status check.
2 Perform the SSH connection check.
3 Repeat the integration procedure.
1.1.2.2 Ports are not open

Resolving the problem when Client is not able to contact with port because ports are
closed.
▪ Open necessary ports on Avamar Client.
For more information about the necessary ports, see the Port Requirements chapter in
Avamar Product Security Guide.
1.1.2.3 Wrong Client name

Resolving the problem when Backup Engine Node is not able to contact with port because
the Client name is wrong.
▪ Ensure that the Client name is properly provided.
<

2. Client integration known warnings
This section describes known warnings, which may apppear after Client integration.
avtar Warning <40010>: High memory consumption.

This is a warning message summarizing the RAM consumption by the avtar process. In any
case, this warning will not cause a crash or indicate an issue with the filesystem.
For more information about managing memory, see the Tuning Performance chapter in Dell
EMC Avamar- Operational Best Practices Guide document.
Errors shown after running a manual backup operation

When manually running a backup, the operation may complete with some errors regarding
attributes (att) present. These errors are created by the Avamar client for every file existing
during the backup operation. Even though the output of the command may produce error
logs, the operation is not disturbed by the event causing them.
Example
2021/04/22-06:06:59.10168 [avtar] ERROR: <0001>
linuxsecurity::init(/private/meta.json) lgetxattr(trusted.glusterfs.crypt.att.size)
failed with errno=61, vallen=-1, bufsize=4096
2021/04/22-06:06:59.10202 [avtar] ERROR: <0001>
linuxsecurity::init(/private/meta.json) lgetxattr(trusted.glusterfs.crypt.att.cfmt)
failed with errno=61, vallen=-1, bufsize=4096


3. Client deleting is failed
This section describes how to troubleshoot when the Client deleting procedure is failed.
Symptoms
It is impossible to delete Client.
Information that operation is failed appears.
Cause
Activities are in progress
3.1 Activities are in progress

Client deletion is failed because other activities are in progress.

Diagnosing the problem when Client deleting is failed because other activities are in
progress.
Purpose
During the Client deletion process, the following log appears:
2020-11-18 11:58:44,990 INFO Deleting client: /ARC/client_hostname
2020-11-18 11:58:45,068 ERROR Client operation failed. Error: Client delete failed.
Cannot delete client.
2020-11-18 11:58:45,069 ERROR Client: /ARC/client_hostname delete procedure has
failed. Error: Client delete failed. Cannot delete client
▪ Check if any activities (backup or restore) for the specified Client are in progress.
<

Resolving the problem when Client deleting is failed because other activities are in
progress.
Procedure
1 If any activity for the desired Client is in progress, wait until the activity is done.
2 Perform the Client deletion operation again. For more information, see the Deleting
Client chapter in Administering Nokia Archive Cloud.


4. Backup operation is failed
This section describes how to troubleshoot when backup operation is failed.
Symptom
The backup operation is not successful.
Causes
Scripts related causes
Connectivity related causes
4.1 Scripts related causes

This section describes how to troubleshoot when backup operation is failed due to scripts
related causes.
Causes
Scripts are not in the correct path
Shell interpreter in scripts is missing
Scripts have no executable rights
4.1.1 Scripts are not in the correct path

Backup operation is failed because scripts are not in the correct path.
4.1.1.1 Diagnosing the problem

Diagnosing the problem when backup operation is failed because scripts are not in the
correct path.
<

Procedure
1 In the Backup Engine Node log (on client in /usr/local/avamar/var/clientlogs
folder or Avamar GUI), one of the following messages appears:
Select from the available options
2017-06-28 21:57:40 avtar Error <7923>: Cannot execute '<script_name>'

script: File not in either path:/usr/local/avamar/etc/scripts/<script_name>
or
path:/usr/local/avamar/scripts/<script_name> (Log #1)
2017-06-28 21:57:40 avtar Error <7001>: Exiting avtar with run-at-start
script failure 1 (Log #1)
or

2017-06-28 21:57:40 avtar Error <7001>: Exiting avtar with run-at-end script
failure 1 (Log #1)
2 Ensure that scripts are in the correct path.
4.1.1.2 Resolving the problem

Resolving the problem when backup operation is failed because scripts are not in the
correct path.
Procedure
1 Move scripts under the /etc/avamar/scripts/ directory.
Note:
Ensure that the script name is correct.
2 Perform the backup operation again. For more information, see the Backup
Management chapter in Administering Nokia Archive Cloud.


4.1.2 Shell interpreter in scripts is missing
Backup operation is failed because shell interpreter in scripts is missing.

Diagnosing the problem when backup operation is failed because shell interpreter in
scripts is missing.
Procedure
2017-06-29 11:36:02 avtar Error <7001>:
Exiting avtar with run-at-start script failure 8 (Log #1)
2017-06-29 11:36:02 avtar Error <7001>:
Exiting avtar with run-at-end script failure 8 (Log #1)
2 Ensure that the shell interpreter is in the script file.

Resolving the problem when Backup operation is failed because shell interpreter in
scripts is missing.
Procedure
1 Add the shell interpreter (#!/bin/bash) at the begging of the script.
2 Save the script.
<

3 Perform the backup operation again. For more information, see the Backup
Management chapter in Administering Nokia Archive Cloud.
4.1.3 Scripts have no executable rights

Backup operation is failed because scripts have no executable rights.

Diagnosing the problem when backup operation is failed because scripts have no
executable rights.
Procedure
or
2017-06-28 22:05:36 avtar Error <7001>: Exiting avtar with run-at-start

script failure 1 (Log #1)
or
2017-06-28 22:05:36 avtar Error <7001>: Exiting avtar with run-at-end script
failure 1 (Log #1)
2 Ensure that the script has an executable rights.


Resolving the problem when backup operation is failed because scripts have no
executable rights.
▪ Provide the executable attribute to the script file:
chmod +x /etc/avamar/scripts/<exec_script.sh>
4.2 Connectivity related causes

This section describes how to troubleshoot when backup operation is failed due to
connectivity related causes.
Causes
No connection to Data Domain
Authentication error occurs in connection to Data Domain
Backups fail with Error Code 10008
4.2.1 No connection to Data Domain

Backup operation is failed because there is no connection to Data Domain.

Diagnosing the problem when there is no connection to Data Domain.
Purpose
In the Backup Engine Node log (on client in /usr/local/avamar/var/clientlogs folder
or Avamar GUI), the following message appears:
<

avtar Info <10539>: Connecting to Data Domain Server
"10.92.53.129"(1) (LSU: avamar-1494422543, User: "********")
avtar Warning <18125>: Calling DDR_OPEN returned result code:5028
message:rpc connection failure
avtar Error <10542>: Data Domain server "10.92.53.129" open failed
DDR result code: 5028, desc: rpc connection failure

Resolving the problem when there is no connection to Data Domain.
Procedure
1 Check if Data Domain is properly configured.
For more information see the Data Domain verification chapter in Administering Nokia
Archive Cloud.
2 Contact Nokia Technical Support for further analysis.
4.2.2 Authentication error occurs in connection to Data

Domain
Backup operation is failed because the authentication error occurs in connection to Data
Domain.

Diagnosing the problem when the authentication error occurs in connection to Data
Domain.
Purpose
or Avamar GUI), the following message appears:


2017-06-20 17:55:26 avtar Info <41236>:
- Connecting to Data Domain Server name "10.92.53.129"
with token:5ba93c9db0cff93f52b521d7420e43f6eda2784f
2017-06-20 17:55:26 avtar Error <41439>:
Using invalid token:5ba93c9db0cff93f52b521d7420e43f6eda2784f
2017-06-20 17:55:26 avtar Error <10542>: Data Domain server "10.92.53.129"
open failed DDR result code: 4904, desc: Invalid API argument.

Resolving the problem when the authentication error occurs in connection to Data
Domain.
Procedure
1 Check if Data Domain is properly configured.
For more information see the Data Domain verification chapter in Administering Nokia
Archive Cloud.
2 Contact Nokia Technical Support for further analysis.
4.2.3 Backups fail with Error Code 10008

This section describes how to troubleshoot when connection with server cannot be
established and backup operation fails with Error Code 10008.

Diagnosing the problem when connection with server cannot be established and backup
operation fails with Error Code 10008.
Purpose
or Avamar AUI), the following message appears:
<

2017-08-11 09:34:58 avtar Info <10539>: Connecting to Data Domain Server
"10.92.53.129"(2) (LSU: avamar-1500977862) with auth token
2017-08-11 09:34:58 avtar Info <10540>: - Resolved Data Domain Server name
"10.92.53.129" to the IP address "10.92.53.129"

2017-08-11 09:34:58 avtar Info <41236>: - Connecting to Data Domain Server name
"10.92.53.129" with token:e160bc7d7ea3c702fc190a7affd92ea7cad62b04
2017-08-11 09:34:58 avtar Info <0000>: - Connecting to:
DataDomain: 10.92.53.129
encryption strength: 2
auth_mode: 2
client_cert /usr/local/avamar/etc/cert.pem
client_key: /usr/local/avamar/etc/key.pem
server_cert:
/tmp/.tmp_avamar/DD_1.0_Test_sto29-1502433280000_598d5010_3c7a708e.pem
chain_cert: /usr/local/avamar/etc/chain.pem
2017-08-11 09:34:58 avtar Info <19155>: - Establishing a connection via token to the
Data Domain system with certificate authentication (Connection mode: A:2 E:2).
2017-08-11 09:35:03 avtar Warning <18125>: Calling DDR_OPEN_VIA_TOKEN returned
result code:5341 message:SSL library error

2017-08-11 09:35:03 avtar Error <10542>: Data Domain server "10.92.53.129" open
failed DDR result code: 5341, desc: SSL library error
2017-08-11 09:35:03 avtar Error <10509>: Problem logging into the DDR server:'',
only GSAN communication was enabled.
2017-08-11 09:35:03 avtar FATAL <17964>: Backup is incomplete because file
"/ddr_files.xml" is missing
2017-08-11 09:35:03 avtar Info <10642>: DDR errors caused the backup to not be
posted, errors=0, fatals=0
2017-08-11 09:35:03 avtar Info <12530>: Backup was not committed to the DDR.
2017-08-11 09:35:03 avtar FATAL <8941>: Fatal server connection problem, aborting
initialization. Verify correct server address and login credentials.
2017-08-11 09:35:03 avtar Info <6149>: Error summary: 4 errors: 10542, 8941, 10509,
17964
2017-08-11 09:35:03 avtar Info <8468>: Sending wrapup message to parent

2017-08-11 09:35:03 avtar Info <5314>: Command failed (4 errors, 1 warning, exit
code 10008: cannot establish connection with server (possible network or DNS
failure))


Resolving the problem when connection with server cannot be established and backup
operation fails with Error Code 10008.
Procedure
1 To retrieve the Data Domain information, perform the following steps:
1.1 Open Avamar AUI, and click System.
1.2 Select the Data Domain.
1.3 Click Edit.
The Edit Data Domain System dialog box appears.
Provide the password in the Password text box and confirm it in Verify
Password.
1.4 Click Validate.
The Get Data Domain system information initiated... message appears.
Note:
There is no need to make any changes to the Data Domain configuration.
The Successfully retrieved Data Domain system information. message appears.
1.5 In the Edit Data Domain window, click Next twice and then Finish.
The It may take a long time to update Data Domain. Please wait... message
appears. Wait until the update is finished.
2 Restart DDBoost on the Data Domain system:
2.1 Log in to the Data Domain System by using CLI.
2.2 Provide the following commands:
<

ddboost disable
ddboost enable
Step result
Connection with the server is established.
4.2.4 Data Domain GUI connection problems

This section describes how to troubleshoot when the user is unable to launch the Data
Domain System Manager from any web browser.

Diagnosing the problem when the user is unable to launch Data Domain System Manager
from any web browser.
Purpose
Service Not Available. The GUI Service is temporarily unavailable. Please refresh
your browser. If the problem persists, please contact DataDomain support for
assistance.
message appears in DD GUI.
This issue can occur when Data Domain system is under heavy load (high
backup/replication/cleaning activity) or if there was a temporary communication issue with
the web service.
DDOS upgrade to 6.x family from 5.6.x or 5.7.x platforms can also cause a connection
problem.
For environments that use self-signed SHA-256 certificates, the certificates must be
regenerated manually after the DDOS upgrade process is complete. Systems that connect
to the Data Domain system must reestablish trust with it.


Resolving the problem when the user is unable to launch Data Domain System Manager
from any web browser.
Purpose
Procedure
1 Close the web browser session with the Data Domain GUI service.
2 Run the following commands in sequence using SSH console:
adminaccess disable http
adminaccess disable https
adminaccess certificate generate self-signed-cert
adminaccess enable http
adminaccess enable https
HTTP and HTTPS services start in a few minutes.
3 Open a web browser and connect to Data Domain System Manager.
Result
The Data Domain GUI is now available.
4.3 Client upgrade related causes

This section describes how to troubleshoot when backup operations are failed due to
upgrade on the Client.

Purpose
After a major upgrade, problems with backup operations may occur on Client side. The
solution is Client reintegration.
<

Purpose
To reintegrate Client, follow the steps below:
Procedure
1 Log in as the brc user to the Management Node on port 7722 using SSH:
ssh brc@<MANAGEMENT_NODE_IP> -p 7722
2 Execute the arccli client reintegrate command, specifying the Client hostname as
follows:
arccli client reintegrate <CLIENT_HOSTNAME>
where:
<CLIENT_HOSTNAME> is a hostname of the integrated Client.
Result
The Client is reintegrated and the following log information is visible:
2018-11-07 14:18:33,694 INFO Reintegrating client: /ARC/client_hostname
2018-11-07 14:18:33,695 INFO Deactivating client: /ARC/client_hostname
2018-11-07 14:18:36,811 INFO Inviting client: /ARC/client_hostname
2018-11-07 14:18:56,848 INFO Client: /ARC/client_hostname has been integrated
4.4 Oracle database backup is failed

This section describes how to troubleshoot when Oracle database backup is failed.
Symptoms


Cause
Timeout on wait for sub-process
First backup of a big database is failed
4.4.1 Timeout on wait for sub-process

Purpose
Oracle backup activity fails with error:
Timeout on wait for sub-process. Please increase subprocess timeout using
--[avoracle]subprocesstimeoutsecs option and try again.

▪ On the Client, create a file /usr/local/avamar/var/avoracle.cmd with the
following content:
--subprocesstimeoutsecs=300
4.4.2 First backup of a big database is failed

Purpose
The first backup of a big Oracle database fails with 24 hours time-out.
<

▪ Optimize RMAN backup parameters according to to the recommendation described
in the Oracle dataset chapter in Nokia Archive Cloud Administering:
channels (rmanChannels)
file per set (backupFilesPerSet)
section size (sectionSize)
4.5 VMware backup is failed
Symptoms
Cause
Datastore space limit is exceeded
VMware backup failed with error No proxy
VMware image backups fails with Error 10056
VM snapshot backup fails due to incorrect pre-evaluation of available space
4.5.1 Datastore space limit is exceeded

Purpose
During backup, 5% of free space is left on the datastore.

▪ Add the additional disk on NFSGlogal1.


4.5.2 VMware backup failed with error No proxy
4.5.2.1 Missing VM proxy (No proxy)
4.5.2.1.1 Diagnosing the problem

List deployed VMware proxies. For more information, see the Listing all Clients chapter in
Administering Nokia Archive Cloud.
4.5.2.1.2 Resolving the problem

Deploy VMware proxies. For more information, see the Deploying VMware proxy chapter in
Administering Nokia Archive Cloud.
4.5.2.2 Missing VM proxy (No proxy)

Purpose
Missing Backup Configuration in VMware proxy client properties.
4.5.2.2.1 Diagnosing the problem

Purpose
Procedure
1 Log in as the brc user to the Management Node on port 7722 using SSH:
ssh brc@<MANAGEMENT_NODE_IP> -p 7722
2 Execute the arccli client show command, specifying the Client hostname as
follows:
<

arccli client show <PROXY_HOSTNAME>
where:
PROXY_HOSTNAME is a hostname of the integrated VMware Proxy.
Result
The proxy details information is visible:
[brc@arc-mgmt-cli ~]$ arccli client show proxy_hostname
2021-01-19 08:46:06,282 INFO Displaying client: /ARC/proxy_hostname details
+------------------------+---------------------------------+
| Attribute | Value |
+------------------------+---------------------------------+
| Client | proxy_hostname |
| Client Domain | /ARC |
| Client Type | VPROXY_REGULAR |
| Enabled | Yes |
| Auto DataStore Mapping | Yes |
| vCenter | vcenter_hostname |
| Max Active Jobs | 8 |
| Backup Configurations | |
+------------------------+---------------------------------+
4.5.2.2.2 Resolving the problem

Enable Auto Proxy Mapping in the Backup Configuration. For more information, see the
Editing Backup Configuration chapter in Administering Nokia Archive Cloud.
4.5.3 VMware image backups fails with Error 10056

This section describes how to troubleshoot when the image backup operation fails with
Error Code 10056.


Diagnosing the problem when backup operation fails with Error Code 10056.
The snapshot create task fails with one the following errors:
2020-06-03 15:09:45 avvcbimage Warning <18652>: Too many extra snapshot files (3)
were found on this Windows VMs datastore. This can cause a problem for the backup or
restore.
2020-06-03 15:09:45 avvcbimage Info <18678>: Extracted VMNAME-flat.vmdk base disk
from the VMX file
2020-06-03 15:09:45 avvcbimage Info <18678>: Extracted VMNAME_1-flat.vmdk base disk
from the VMX file
2020-06-03 15:09:45 avvcbimage FATAL <0000>: [IMG0002] The datastore information from
VMX '[DATASTORENAME01] VMNAME/VMNAME.vmx' will not permit a restore or backup.
2020-06-03 15:09:45 avvcbimage Info <9772>: Starting graceful (staged) termination,
Too many pre-existing snapshots on the datastore will not permit a restore or backup.
(wrap-up stage)
2020-06-03 15:09:45 avvcbimage Error <0000>: [IMG0009] Create Snapshot: snapshot
creation or pre/post snapshot script failed.
2020-06-03 15:09:45 avvcbimage Error <0000>: [IMG0009] Create Snapshot: snapshot
creation/pre-script/post-script failed.

Resolving the problem when the backup operation fails with Error Code 10056.
Before you start
Ensure there are no snapshots in vSphere - Snapshot manager and/or Unconsolidated

Snapshots.
Purpose
To resolve the problem, follow the steps below:
Procedure
1 Look for avvcbimage Info <14677> in the logs and verify if there are files like
<

VMNAME-000002-delta.vmdk, VMNAME-000002-ctk.vmdk, or VMNAME-000002-
sesparse.vmdk.
In this example VMNAME-00000X-ctk.vmdk and VMNAME-00000X-sesparse.vmdk

files were found:
2020-06-03 15:09:44 avvcbimage Info <14677>: All VM file Info:
File:'VMNAME-000002-ctk.vmdk
File:'VMNAME-000002-sesparse.vmdk
File:'VMNAME-000002.vmdk
File:'VMNAME-766073b0.hlog
...
File:'vmware-50.log
File:'vmware-51.log
File:'vmware-52.log
File:'vmware.log
File:'vmx-VMNAME-3818966866-1.vswp
File:'vmx-zdump.000
2020-06-03 15:09:44 avvcbimage Info <19704>: DataStore Storage
Info:DATASTORENAME01 capacity=2465042792448 free=349293248512


' last modified on '29-Mar-2020 21:21' size=3932672
' last modified on '12-Apr-2020 20:46' size=3932672
' last modified on '05-May-2020 19:41' size=92
' last modified on '02-Jun-2020 14:20' size=306251
2 The easiest and fastest way to fix this issue is doing a Datastore vMotion.
3 After this action, try a new backup.
4 If it still fails, verify if the Datastore has changed, looking for avvcbimage Info
<19704> in the current and previous log:
5 After the Datastore vMotion in the log, the file date modification must be updated
for all files, and the additional used files were cleaned.
<

2020-06-03 17:42:58 avvcbimage Info <14677>: All VM file Info:
File:'TSPEGASUS-682f3bfa.hlog
File:'TSPEGASUS-aux.xml
File:'TSPEGASUS-ctk.vmdk
File:'TSPEGASUS-e40ee848.vswp
File:'TSPEGASUS-flat.vmdk
File:'TSPEGASUS.nvram
File:'TSPEGASUS.vmdk
File:'TSPEGASUS.vmsd
File:'TSPEGASUS.vmx
File:'TSPEGASUS.vmx.lck
File:'TSPEGASUS.vmxf
File:'TSPEGASUS.vmx~
File:'TSPEGASUS_1-ctk.vmdk
File:'TSPEGASUS_1-flat.vmdk
File:'TSPEGASUS_1.vmdk
File:'vmmcores-1.gz
File:'vmware-0.log
File:'vmware-48.log
File:'vmware-49.log
File:'vmware-50.log
File:'vmware-51.log
File:'vmware-52.log
File:'vmware.log
File:'vmx-TSPEGASUS-3826182216-1.vswp
File:'vmx-zdump.000


6 If it still fails, open a support case.
4.5.4 VM snapshot backup fails due to incorrect pre-

evaluation of available space
This section describes how to troubleshoot when VM snapshot backups fails due to
incorrect pre-evaluation of available space.
The snapshot_max_change_percent parameter tells the proxy to pre-evaluate free

datastore space to ensure that there is enough storage for the VM snapshot. The default
<

value is set to 5%.

Purpose
There are two options to solve the issue:
1. Add an additional space on NFSGlobal1.

2. To permanently override this check in the proxy, log in to each proxy and modify the file
/usr/local/avamarclient/avvcbimageAll.cmd to include the line:
-- snapshot_max_change_percent=0.


5. Alarms are raised in OMAgent
This section describes how to troubleshoot when alarms are raised in OMAgent.
5.1 Alarm 2004750: Datadomain get checkpoint list

operation failed is raised
This section describes how to troubleshoot when Datadomain get checkpoint list
operation failed alarm is raised.
Symptoms:
An alarm is raised on the monitoring server:
2004750 - Datadomain get checkpoint list operation failed
Causes:
Password for ddboost user has expired.
5.1.1 Password for ddboost user expired

Diagnosing the problem when the password for ddboost user expired.
Purpose
▪ Log in to the Data Domain as sysadmin user and execute:
user show list
<

Step result
Status of the ddboost user (the exemplary user: boost-ci) islocked.
User list from node "localhost".
Name Uid Role Last Login From
-------------- --- --------------- ---------------
sysadmin 100 admin 10.154.190.19
boost-ci 508 admin 10.40.32.206
Last Login Time Status Disable Date
------------------------ ------- ------------
Wed Nov 28 14:08:54 2018 enabled never
Wed Nov 28 14:08:27 2018 locked never

Resolving the problem when password for ddboost user expired.
Purpose
The password for the ddboost user must be changed within the number of days specified
by the password policy. When the password has not been changed, the ddboost user is
locked.
Note:
Before starting this procedure, check the Data Domain IP and a password for an
admin user on the Backup Engine Node (Avamar). It is recommended to log in on the
Backup Engine Node as an admin user via ssh on a separate console.
There is a time limitation up to 10 minutes to finish the procedure. After this time,
when the procedure is not finished, ddboost user will be blocked.
To unlock and change the password for ddboost user, execute the following steps:
Procedure
1 Log in to the Data Domain as sysadmin user and execute:


user show list
Step result
Status of the ddboost user (the exemplary user: boost-ci) is locked.
-------------- --- --------------- ---------------

boost-ci 508 admin 10.40.32.206
------------------------ ------- ------------

Wed Nov 28 14:08:27 2018 locked never
2 To check number of maximum login attempts, execute:
adminaccess option show login-max-attempts
Step result
Number of maximum attempts appears.
Adminaccess option "login-max-attempts" set to "4".
3 To change the number of maximum login attempts to 20, execute:
adminaccess option set login-max-attempts 20
4 To change password for the ddboost user, execute:
user change password <DdBoostUsername>
where:
DdBoostUsername is a name of the ddboost user (the exemplary user: boost-ci).
<

5 Enter the new password according to the password policy.
Note:
You need to enter a new password twice.
Step result
Passwords matched.
Password changed for user "boost-ci".
6 After log in to the Backup Engine Node (Avamar) as the admin user, execute:
mccli dd edit --password=<NewDdBoostUserPassword> --password-
confirm=<NewDdBoostUserPassword> --name=<DataDomainIp>
where:
NewDdBoostUserPassword is a new password for the ddboost user.
DataDomainIp is an IP address of Data Domain.
Step result
0,31005,Data Domain system updated but the hostname may not be valid.
Attribute Value
------------ ------------
dd 10.41.119.63
hostname 10.41.119.63
ipv6Hostname
ipv4Hostname 10.41.119.63
7 Log in as the sysadmin user or switch back to the Data Domain console and execute:
user show list
Step result
Status of the ddboost user (the exemplary user: boost-ci) is enabled.


-------------- --- --------------- ---------------

boost-ci 508 admin 10.40.32.206
------------------------ ------- ------------

8 To revert the number of maximum login attempts, execute:
adminaccess option set login-max-attempts <MaxLoginAttemps>
where:
MaxLoginAttemps is a previously number of Maximum Login Attempts, for example 4.
9 Clear alarm manually from the monitoring system.
<

6. Performance Managemenet data is delayed in NetAct
This section describes how to troubleshoot when Performance Management data from
Archive Cloud is delayed in NetAct.
6.1 Performance Management data collection schedule

interval is too short
OMAgent's Performance Management (PM) data collection does not fit into the timeframe
set by the schedule interval, causing NetAct to receive delayed PM data from Archive Cloud
(ARC).

Diagnosing the problem when collection schedule interval is too short.
Purpose
PM data collection task takes more time than the schedule interval is set to, which results in
NetAct receiving delayed PM data.

▪ Increase the collection schedule interval so it is longer than the actual collecting
time.
For information on how to change the interval, see the Changing OMAgent PM data
collection schedule interval chapter in Administering Nokia Archive Cloud.

Troubleshooting Nokia Archive Cloud

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Troubleshooting Nokia Archive Cloud

Uploaded by

Copyright:

Available Formats

No further reproduction or networking is permitted. Distributed by Nokia.

Copyrighted material licensed to khan.tanveer.ext@nokia.com on 12-06-2022.