Restricted View

Knowledge Base Article: 000457974

Data Domain: How to upload a core dump file from a Data Domain Restorer for analysis by Data
Domain support/engineering (000457974)
Primary Product : Data Domain
Product : Data Domain,DD OS

Version: 6 Article Type: Break Fix Audience: Level 30 = Customers Last Published: Fri Dec 02 12:59:01 GMT 2016

This article describes the steps required to verify if a process or kernel core dump file has been generated by an unexpected
Summary: event on a Data Domain Restorer and if so how to locate the corresponding file and upload for analysis by Data Domain
support/engineering

Version: 6 Article Type: Break Fix Audience: Level 30 = Customers Last Published: Fri Dec 02 12:59:01 GMT 2016

Summary: This article describes the steps required to verify if a process or kernel core dump file has been generated by an unexpected
event on a Data Domain Restorer and if so how to locate the corresponding file and upload for analysis by Data Domain support/engineering
Issue:
Data Domain Restorers (DDRs) run a linux based operating system known as the Data Domain Operating System
(DDOS). DDOS is configured such that if a process within DDOS (or the DDOS kernel itself) experiences an issue
causing unexpected termination (i.e. the process or kernel crashes) then a corresponding core dump file will be created.
Note that creation of core dump files can also be forced by Data Domain support when investigating certain issues.

A core dump file contains an image of the working memory being used by the process or kernel at the point of termination
and can be invaluable for post mortem analysis. This article describes:

How a core dump file is created

How to determine which core dump files exist on a DDR
How to upload a core dump file for further analysis by Data Domain support/engineering

Cause:
If a process running on a DDR terminates unexpectedly (crashes) then:

The process will be frozen

An image of the working memory of the process will be dumped to a dedicated area of disk (the dump device)
The process will exit and can then be restarted (in most cases this takes place automatically)
In parallel to the process restarting dumped information is read from the dump device and written to a compressed core
dump file
Whilst being initially written the core dump file will be named 'core-imcomplete.gz' and is staged to a temporary directory
named /ddr/var/core/app-[YYYYMMDD]-[HH.MM], for example:

/ddr/var/core/app-20160902-04.05:
-rw-r--r-- 1 root root 32538624 Sep 2 04:05 core-incomplete.gz

Once the dump has completed the core dump file will be moved to the /ddr/var/core directory, for example:

-rw-r--r-- 1 root root 87980106 Sep 2 04:06

/ddr/var/core/ddfs.core.25935.1472814293.gz

Note that:

The uncompressed size of the core dump file is directly proportional to the amount of memory being used by the process
at the time of termination - as a result core dump files from some processes such as 'DDFS' (the Data Domain File
System) may be close to the amount of RAM installed in the DDR in size. Even once compressed these core dump files
may still be many tens of Gb in size - this is expected
It will take time for the contents of a terminating processes memory to be written to the dump device - again for large
processes this can take a number of minutes. As a result there may be a short delay before the terminating process exits
and can be restarted/service resumed
A core dump file may not be generated after every unexpected termination of a process - this depends on there being
sufficient space to write the compressed core dump file in the /ddr/var/core directory and the type of issue encountered
Compressed core dump files may be written to a location other than /ddr/var/core, for example, on larger systems core
dump files may actually exist under /ddr/var/ext - if this happens, however, there will be a symbolic link created in the
/ddr/var/core directory pointing to the physical core dump file so all information in this article remains valid/correct
 The name of the core dump file contains useful information which can be used to identify the process which created the
dump and the time/date of creation, i.e.:
${PROCESS_NAME}.core.${PID}.${UNIX_EPOCH}.gz

Where:

PROCESS_NAME: the name of the process that terminated and generated this core dump file
PID: the process ID which was being used by the process at the point of termination
UNIX_EPOCH: time in seconds since January 1 1970 when the process core dump was created

If the DDOS kernel terminates unexpectedly (crashes) then:

The operating system will be halted
An image of the working memory of the DDOS kernel will be dumped to a dedicated area of disk (the dump device)
The system will be automatically rebooted
During boot DDOS will check the dump device and find that it contains a valid 'kernel dump'
In parallel to the system booting the kernel dump will be read from the dump device and written to a compressed core
dump file
The core dump file will be located in its own directory under /ddr/var/core and will always be named vmcore.gz, for
example:
/ddr/var/core/kernel-20141125-16.50:
-rw-r--r-- 1 root admin 102603347 Nov 25 2014 vmcore.gz

Note that:
It will take time for the working memory of the DDOS kernel to be dumped to disk - as a result there may be a short delay
before the system reboots
A core dump file may not be created after every unexpected termination of the DDOS kernel - again this depends on there
being sufficient space to write the compressed core dump file to the /ddr/var/core directory and the type of issue
encountered
The name of the directory containing the core dump file indicates the date/time at which the core dump was created as
this is always named kernel-[YYYYMMDD-HH.MM]

Resolution:
Step 1 - Identify core dump files on the DDR
It is first necessary to identify which core dump files exist on a DDR - this can be done by listing the contents of the
/ddr/var/core directory.

Identifying core dump files on the DDR itself - DDOS 5.5.x (and later):

Log into the Data Domain command line shell (DDSH)

Run the 'support coredump list' command - this lists the contents of the /ddr/var/core directory:

# support coredump list

File Size Time Created
------------------------------------- --------- ------------------------
ddfs.core.11539.1460540776.gz 470.1 MiB Wed Apr 13 02:49:33 2016
ddfs.core.16431.1456839840.gz 117.7 MiB Tue Mar 1 05:45:21 2016
ddfs.core.2088.1460394487.gz 39.2 MiB Mon Apr 11 10:09:00 2016
sms.core.5335.1458725501.gz 4.8 MiB Wed Mar 23 02:31:42 2016
------------------------------------- --------- ------------------------

Note that this command only lists process core dump files and does not provide details of kernel core dump files

Identifying core dump files on the DDR itself - all versions of DDOS:

Log into the Data Domain command line shell (DDSH)

Drop to 'se' mode:

# system show serialno

[system serial number displayed]
# priv set se
[password prompt - enter system serial number from above]

Note that on some systems (such as those using encryption of data at rest) the 'priv set se' command may also
prompt for details of a user with a role of 'security'

List the contents of the /ddr/var/core directory using the 'se ls -l /ddr/var/core' command:

# se ls -l /ddr/var/core
total 647612
-rw-r--r-- 1 root root 492903386 Apr 13 02:49 ddfs.core.11539.1460540776.gz
-rw-r--r-- 1 root root 123426535 Mar 1 2016 ddfs.core.16431.1456839840.gz
-rw-r--r-- 1 root root 41100854 Apr 11 10:09 ddfs.core.2088.1460394487.gz
drwx------ 2 root root 4096 Sep 6 12:20 kernel-20160906-01.14
drwx------ 2 root root 16384 Feb 23 2016 lost+found
-rw-r--r-- 1 root root 488 Sep 6 01:15 savecore_log.20160906-01.14
-rwxrwxrwx 1 root root 5021185 Mar 23 02:31 sms.core.5335.1458725501.gz

Note that this lists all process core dump files and any directories containing kernel core dump files. To list the
contents of a directory containing a kernel core dump file use 'se ls -l /ddr/var/core/[directory]':
 # se ls -l /ddr/var/core/kernel-20160906-01.14
total 201516
-rw-r--r-- 1 root root 206141752 Sep 6 01:15 vmcore.gz

Identifying core dump files using an external FTP/FTPS client:

Confirm that the FTP/FTPS service is enabled on the DDR and that the client system is an 'allowed host' (refer to the
'notes' section of this article for details)
Connect to the DDR using the FTP/FTPS client - note that the username used to establish the connection must have a
role of admin on the DDR:

$ ftp 10.64.13.192
...
Name (10.64.13.192:fordj6): sysadmin
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

List the contents of the 'core' directory:

ftp> ls core
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
-rw-r--r-- 1 0 0 492903386 Apr 13 09:49
ddfs.core.11539.1460540776.gz
-rw-r--r-- 1 0 0 123426535 Mar 01 2016
ddfs.core.16431.1456839840.gz
-rw-r--r-- 1 0 0 41100854 Apr 11 17:09
ddfs.core.2088.1460394487.gz
drwxr-xr-x 2 0 0 4096 Sep 08 08:48 kernel-20160906-01.14
drwx------ 2 0 0 16384 Feb 23 2016 lost+found
-rw-r--r-- 1 0 0 488 Sep 06 08:15 savecore_log.20160906-01.14
-rwxrwxrwx 1 0 0 5021185 Mar 23 09:31 sms.core.5335.1458725501.gz
226 Directory send OK.

The contents of directories containing kernel core dump files can be listed by appending the directory name:

ftp> ls core/kernel-20160906-01.14
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
-rw-r--r-- 1 0 0 206141752 Sep 06 08:15 vmcore.gz
226 Directory send OK.

Identifying core dump files using an external NFS client:

Confirm that the NFS protocol is enabled on the DDR and that an export for /ddvar/core exists and the NFS client system
has access (refer to the 'notes' section of this article for details)
On the NFS client mount the /ddvar/core directory from the DDR:

root@ubuntu-jamesf:~# mkdir /root/mnt

root@ubuntu-jamesf:~# mount 10.64.13.192:/ddvar/core /root/mnt
root@ubuntu-jamesf:~#

List the contents of the mounted /ddvar/core directory (and any subdirectories containing kernel core dump files):

root@ubuntu-jamesf:/# ls -l /root/mnt
total 5180896
-rw-r--r-- 1 root root 492903386 Apr 13 10:49 ddfs.core.11539.1460540776.gz
-rw-r--r-- 1 root root 123426535 Mar 1 2016 ddfs.core.16431.1456839840.gz
-rw-r--r-- 1 root root 41100854 Apr 11 18:09 ddfs.core.2088.1460394487.gz
drwxr-xr-x 2 root root 4096 Sep 8 09:48 kernel-20160906-01.14
drwx------ 2 root root 16384 Feb 23 2016 lost+found
-rw-r--r-- 1 root root 488 Sep 6 09:15 savecore_log.20160906-01.14
-rwxrwxrwx 1 root root 5021185 Mar 23 09:31 sms.core.5335.1458725501.gz
root@ubuntu-jamesf:/# ls -l /root/mnt/kernel-20160906-01.14/
total 1612128
-rw-r--r-- 1 root root 206141752 Sep 6 09:15 vmcore.gz

Identifying core dump files using an external CIFS client:

Confirm that the CIFS protocol is enabled on the DDR and that a share for /ddvar exists and the CIFS client system has
access (refer to the 'notes' section of this article for details)
Use the CIFS client to browse the contents of the 'core' directory within the CIFS share:
Identifying core dump files using an autosupport from the system:

Autosupports contain a listing of the contents of the /ddr/var/core directory - as such any autosupport created after a core
dump file is created will contain details of that file. For example:

/ddr/var/core:
total 5180896
-rw-r--r-- 1 root root 492903386 Apr 13 02:49 ddfs.core.11539.1460540776.gz
-rw-r--r-- 1 root root 123426535 Mar 1 2016 ddfs.core.16431.1456839840.gz
-rw-r--r-- 1 root root 41100854 Apr 11 10:09 ddfs.core.2088.1460394487.gz
drwx------ 2 root root 4096 Sep 6 12:20 kernel-20160906-01.14
drwx------ 2 root root 16384 Feb 23 2016 lost+found
-rw-r--r-- 1 root root 488 Sep 6 01:15 savecore_log.20160906-01.14
-rwxrwxrwx 1 root root 5021185 Mar 23 02:31 sms.core.5335.1458725501.gz

/ddr/var/core/kernel-20160906-01.14:
total 201524
-rw-r--r-- 1 root root 206141752 Sep 6 01:15 vmcore.gz

Step 2 - Determine which core files are required for further analysis:
Commonly a core dump file will be associated with a single event which took place on the DDR, i.e.:

A single unexpected restart of a process - in this case there may be a single core dump file within the core directory
relating to the restart
A single unexpected reboot of the system - in this case there may be a single 'kernel-[YYYYMMDD-HH.MM]' directory
within the core directory relating to the reboot - within this directory there will be a single vmcore.gz file

Verify that the date/time of the corresponding file matches the approximate date/time of the issue and that it is of the
expected type (for example a restart of the DDFS process may produce a core dump file with 'ddfs' in its name, a reboot
of the system may produce a core dump file called vmcore.gz).

In this scenario this single file should be uploaded for further analysis.

In some scenarios, however, the issue may have caused multiple core dump files to be created - for example the DDFS
process may have unexpectedly terminated multiple times in a short duration creating multiple core dump files. In this
case look for the oldest core dump file created during the issue (i.e. the first in the series of corresponding core dump
files). Initially this single file should be uploaded for further analysis - Data Domain support will then advise whether further
core dump files are required.

Step 3 - Collect the core dump file from the DDR:

Before the core dump file can be uploaded to Data Domain support for further analysis it needs to be copied off of the
DDR onto a client system with access to the internet. This can be achieved in a number of ways:

Via FTP/FTPS:

Confirm that the FTP/FTPS service is enabled on the DDR and that the client system is an 'allowed host' (refer to the
'notes' section of this article for details)
Connect to the DDR using the FTP/FTPS client - note that the username used to establish the connection must have a
role of admin on the DDR:

$ ftp 10.64.13.192
...
Name (10.64.13.192:fordj6): sysadmin
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
Change to the directory containing the core dump file to be uploaded:

ftp> cd core/kernel-20160906-01.14
250 Directory successfully changed.

Set transfer mode to binary:

ftp> bin
200 Switching to Binary mode.

Download the file to the client system:

ftp> get vmcore.gz

local: vmcore.gz remote: vmcore.gz
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for vmcore.gz (206141752 bytes).
226 Transfer complete.
206141752 bytes received in 0.87 secs (230237.1 kB/s)

Via NFS:

Confirm that the NFS protocol is enabled on the DDR and that an export for /ddvar/core exists and the NFS client system
has access (refer to the 'notes' section of this article for details)
On the NFS client mount the /ddvar/core directory from the DDR:

root@ubuntu-jamesf:~# mkdir /root/mnt

root@ubuntu-jamesf:~# mount 10.64.13.192:/ddvar/core /root/mnt
root@ubuntu-jamesf:~#

Copy the core dump file from the NFS mounted /ddvar/core directory to a local directory:

root@ubuntu-jamesf:/# cp /root/mnt/kernel-20160906-01.14/vmcore.gz /tmp

root@ubuntu-jamesf:/#

Via CIFS:

Confirm that the CIFS protocol is enabled on the DDR and that a share for /ddvar exists and the CIFS client system has
access (refer to the 'notes' section of this article for details)
Browse to the CIFS share and copy the core dump file to a local directory

Via SCP:

Confirm that the SCP service is enabled on the DDR:

# adminaccess show
Service Enabled Allowed Hosts
----------- ------- -------------
ssh yes -
scp no (same as ssh)
...

If necessary enable the SCP service:

# adminaccess enable scp

SCP Access: enabled

Use the SCP client to copy the core dump file to the local system:

root@ubuntu-jamesf:/# scp
sysadmin@10.64.13.192:/ddr/var/core/kernel-20160906-01.14/vmcore.gz /tmp
EMC Data Domain Virtual Edition
Password:
vmcore.gz 100% 197MB 98.3MB/s 00:02

Step 4 - Upload the core dump file for further review:

When uploading a core dump file for further review it is expected that:
A support bundle has been collected on the DDR after the core dump file was created and has either already been
uploaded to EMC Data Domain support or will be uploaded along side the core dump file - for details of creating a support
bundle see the following KB article: https://support.emc.com/kb/323283
The core dump file (and possibly support bundle) is available on a client system with access to the internet
A corresponding service request is already open with EMC Data Domain support and that the service request number is
know
The core dump file can then be uploaded in a number of ways:

Temporary FTP account:

An EMC support engineer can provide details of a temporary FTP account which remains available for use for 7 days
from creation. This temporary account can be accessed via the following clients:
A web browser (using the HTTPS protocol)
FTP/SFTP
To obtain details of a temporary FTP account please contact the EMC support engineer dealing with any corresponding
service request

Anonymous FTP:
 Use an FTP client to connect to ftp.emc.com and log in as 'anonymous' (note that your e-mail address should be used as
the password):
root@ubuntu-jamesf:/tmp# ftp ftp.emc.com
...
Name (ftp.emc.com:fordj6): anonymous
331 User name okay, need password.
Password:
230 User logged in, proceed.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
Change directory to /incoming/datadomain:
ftp> cd /incoming/datadomain
250 File action OK.

Note that the /incoming/datadomain directory is blind (i.e. its contents cannot be listed - this is expected)
Create a new directory with name equivalent to the corresponding service request:
ftp> mkdir 12345678
257 "/incoming/datadomain/12345678"
Enter the newly created directory:
ftp> cd 12345678
250 File action OK.
Set transfer mode to binary:
ftp> bin
200 Command OK.
Use the 'put' command to upload the core dump file:
ftp> put vmcore.gz
local: vmcore.gz remote: vmcore.gz
200 Command OK.
150 File OK.
226 /incoming/datadomain/12345678/vmcore.gz
206141752 bytes sent in 315.77 secs (637.5 kB/s)

Upload to existing service request:

Use a web browser to access the existing service request - note that the URL of the service request may change
depending on whether the Service Center or Service Cloud portals are being used
Click on the icon to attach/upload a file to the service request
Follow the on screen prompts to complete upload of the file
Note that certain portals may have limits on the size of files which can be attached - in this case the core dump file should
instead be uploaded via temporary FTP account/anonymous FTP

Note that if core dump files are extremely large it may be beneficial to split them prior to upload such that the upload can
be completed in smaller increments. For details on splitting a file prior to upload please refer to the following article:
https://support.emc.com/kb/333893

Step 5 - Notify EMC Data Domain support that the files have been uploaded:
Regardless of method used to upload the core dump file (and possibly support bundle) the EMC Data Domain support
engineer dealing with you service request should be notified that upload is complete. Please specify:
Upload method used
Names/sizes of files uploaded
If using anonymous FTP the exact directory created on ftp.emc.com to which files were uploaded
If the core dump file was split the method used to achieve this
Failure to notify the support engineer may cause delays in further analysis.

Notes: How to enable the FTP/FTPS service on a DDR and provide access to client systems:
Use the 'adminaccess show' command to determine whether the FTP/FTPS services are enabled and which client
systems are currently specified as 'allowed hosts':
# adminaccess show
Service Enabled Allowed Hosts
----------- ------- -------------
...
ftp no -
ftps no -
...
----------- ------- -------------
To enable a specific service use the 'adminaccess enable' command, i.e.:
# adminaccess enable ftps
FTPS Access: enabled

Note that FTP and FTPS are mutually exclusive - as a result it may be necessary to disable an existing enabled
service before the required service can be enabled:

# adminaccess enable ftp

Enabling FTP Failed: Disable FTPS before enabling FTP
# adminaccess disable ftps
FTPS Access: disabled
# adminaccess enable ftp
FTP Access: enabled
To allow FTP/FTPS access to the DDR by a specific client system the following command should be used (note that the
required service should be specified as necessary):
 # adminaccess ftp add 10.64.13.191
Security access lists updated
The ftp access list is:
10.64.13.191

Alternatively to allow FTP/FTPS access to the DDR by all clients the following command should be used (again
the required service should be specified as necessary):

# adminaccess ftp add *

Security access lists updated
The ftp access list is:
*

Note that specific client systems can be identified using host name or IP address however if the host name is
used this must be resolvable by the DDR.

How to enable NFS access to /ddr/var/core on DDR and provide access to client systems:
Confirm that the NFS protocol is enabled on the DDR:
# nfs enable
NFS server is already enabled.
Verify whether the /ddvar/core directory is already NFS exported and if so whether the corresponding NFS client has
access (in this case the directory is not NFS exported):

# nfs show clients

Path Client Options
----------------- ------ ------------------------------------------------------
/data/col1/backup * (sec=sys,rw,no_root_squash,no_all_squash,secure,nolog)
----------------- ------ ------------------------------------------------------

If no NFS export for /ddvar/core exists or the NFS client does not have access to an existing export a new export should
be created
To allow access to the /ddvar/core export for a specific client system the following command can be used:

# nfs add /ddvar/core 10.64.13.191

NFS export for "/ddvar/core" added.

Alternatively to allow access to the /ddvar/core export from all client systems the following command should be
used:

# nfs add /ddvar/core *

NFS export for "/ddvar/core" added.

Note that specific client systems can be identified using host name or IP address however if the host name is
used this must be resolvable by the DDR.

How to enable CIFS access to /ddr/var/core on DDR and provide access to client systems:
Confirm that the CIFS protocol is enabled on the DDR:
# cifs enable
The filesystem is enabled and running.
Starting CIFS access...
Verify whether a CIFS share already exists for the /ddvar directory and if so whether the corresponding CIFS client has
access (in this case the directory is not shared via CIFS):

# cifs share show

Shares information for: all shares

--------------- share backup ---------------

path: /data/col1/backup
clients: *
enabled: yes

Shares displayed: 1

If no CIFS share exists for the /ddvar directory a new share will need to be created. This can be done specifying the
explicit host name/IP address of the CIFS clients:

# cifs share create ddvar path /ddvar clients 10.66.59.229

Share "ddvar" has been created.

Alternatively access can be provided for all CIFS clients:

# cifs share create ddvar path /ddvar clients *

Share "ddvar" has been created.

If an existing CIFS share exists for the /ddvar directory however the CIFS client is not listed against the share the client
will need to be added:

# cifs share modify ddvar clients 10.66.59.229 clients 10.64.13.191

Share "ddvar" is modified for new connections.

Note that:
 All of the above examples assume that required network ports are open between the client system and the DDR - for
details of ports used see the following KB article: https://support.emc.com/kb/323297
Any changes made to the DDR whilst configuring FTP/FTPS/NFS/CIFS access should be reversed once core dump files
have been identified and copied off of the DDR
For further details around using CIFS when a DDR is configured in workgroup mode see the following KB article:
https://support.emc.com/kb/323398
There may be some scenarios where permissions set against core dump files generated on a DDR prevent access by
users and as a result they cannot be listed/copied off of the system. In this scenario permissions against files can be
opened up as follows:
Log into the Data Domain command line shell (DDSH)
Drop to 'se' mode:

# system show serialno

[system serial number displayed]
# priv set se
[password prompt - enter system serial number from above]

Note that on some systems (such as those using encryption of data at rest) the 'priv set se' command may also
prompt for details of a user with a role of 'security'
Open up restrictions on the corresponding file/directory using the 'se chmod' command, for example:

# se chmod -R 755 /ddr/var/core/kernel-20160906-01.14

Files should then be able to be accessed with no issues

Primary Product: Data Domain

Product: Data Domain,DD OS

Problem Code: Corrective Maintenance

Article Properties: Validation Status: Approved Original Create Date: Wed Nov 30 11:35:45 GMT 2016

Channels: First Published: Fri Feb 05 17:07:43 GMT 2016

Originally Created By: SFDC Admin Last Modified: Fri Dec 02 12:59:01 GMT 2016