Data Collection Guideline

Submitting a Customer Service Request

SYSTEM ADMIN GUIDE

92/1543-AXI 101 09/1 Uen B

Copyright

© Ericsson AB 2016. All rights reserved. No part of this document may be

reproduced in any form without the written permission of the copyright owner.

Disclaimer

The contents of this document are subject to revision without notice due to
continued progress in methodology, design and manufacturing. Ericsson shall
have no liability for any error or damage of any kind resulting from the use
of this document.

92/1543-AXI 101 09/1 Uen B | 2016-09-28

 Contents

Contents

1 Introduction 1

2 Workflow 1

3 Mandatory Data 1
3.1 Mandatory Logs and Data 3
3.2 Acceptable Compression Formats 5

4 Collecting Data Based on Problem Type 5

5 Severity Levels 7
5.1 Emergency 7
5.2 High Severity 8
5.3 Medium Severity 8
5.4 Low Severity 9

Glossary 10

92/1543-AXI 101 09/1 Uen B | 2016-09-28

Data Collection Guideline

92/1543-AXI 101 09/1 Uen B | 2016-09-28

 Mandatory Data

1 Introduction

This document provides information on the troubleshooting data that must be

collected and enclosed in a Customer Service Request (CSR) if you encounter
problems with Ericsson Router 6672. This document also describes the severity
levels for a CSR and the procedure for collecting the necessary information.

This document is intended for experienced Operation and Maintenance (O&M)

personnel troubleshooting the router.

2 Workflow

The workflow for collecting troubleshooting data for a CSR is as follows:

1 Collect the general mandatory data for the problem experienced. See
Section 3 on page 1.

2 Collect specific data based on the problem type. See Section 4 on page 5.

3 State the severity level of the CSR. See Section 5 on page 7.

3 Mandatory Data

You must collect the following data for a CSR, regardless of the problem type.

Use the checklist in Table 1 when writing a CSR. A description of each line
follows the checklist. Some parts are not applicable to certain types of requests
such as documentation errors or consultations, but all parts apply to software
and hardware faults. Complete as many checklist items as possible for each
case.

92/1543-AXI 101 09/1 Uen B | 2016-09-28 1

 Data Collection Guideline

Table 1 Checklist for Writing a CSR

release:
HW platform:

RECENT CHANGES
SW changes:
Configuration changes:
HW changes:
O&M procedures:
Environment:

CASE DESCRIPTION
Case description:
Date and time:
Problem frequency:
Problem reproducibility:
Problem effects:
Network diagram to illustrate the problem:

ATTACHMENTS
System logs:
Crashfiles:
Output of the show tech-support command:
Network Diagram/Topology:
Other attachments:

MEASURES
On-site or online support:
Work around:

• Release: Specify the release.

• Hardware Platform: Specify the hardware platform on which the fault

occurred. If known, specify the device and card where the fault was found;
for example, specify the chassis serial number from the output of the show
hardware command or the slot for the card.

• Recent Changes: Specify if any of the following changes have been

implemented or occurred recently:

0 Software updates or upgrades

0 Board replacement or hardware upgrades

0 Changes in upgrade, update, or installation procedures

0 Configuration changes

0 Changes in the network (for example, a cutover)

0 Any other relevant changes

2 92/1543-AXI 101 09/1 Uen B | 2016-09-28

 Mandatory Data

• Case Description: To make it easier for Ericsson support personnel to

locate the problem in the log files, provide a clear description of the
consultation or the problem.

0 Describe the case by answering the following questions:

• What is the end-subscriber impact of the problem?

• What is the network impact of the problem?

• What is the operational limitation of the problem?

0 State the exact date and time when the fault occurred or was
discovered.

0 Specify the frequency of the problem: Is it a one-time or an occasional

fault? If it occurs frequently, how often?

0 Specify the reproducibility of the problem.

• If the fault is reproducible, attach a step-by-step description of how

to reproduce the fault or under which circumstances the problem
occurs.

• If the fault is non-reproducible, describe the circumstances before

and after the fault. For example, core dump generated.

0 Describe the effects of the fault; for example, whether it affected traffic
or O&M.

0 If helpful, include a logical picture of the network diagram showing

the direct environment of the affected node; for example, interfaces to
which the node is directly connected.

• Attachments: Provide the filename and format of all attachments. Always

include the mandatory files as described in Section 3.1 on page 3 and,
when necessary, include data described in Section 4 on page 5.

• Measures:

0 Describe if it is possible to stabilize the node by manual intervention;

for example, with a reload.

0 Provide a description of a possible workaround and the related

commands.

3.1 Mandatory Logs and Data

Always include the following logs and data in the CSR.

92/1543-AXI 101 09/1 Uen B | 2016-09-28 3

 Data Collection Guideline

System Log Files from the Controller Card

Collect system logs located in the /var/log directory through following

Command-Line Interfaces (CLIs) in Linux shell mode either to copy var logs to
flash or md:

tar -zcvf/flash/varlog.tar.gz /var/log

tar –zcvf/md/varlog.tar.gz /var/log

The log file must include the time of the failure. In the CSR, also include the
time stamps before and after the event occurred. It is important to verify which
file contains the failure, because the active message log file is eventually
overwritten. For example, the file can be /var/log/messages.2.gz instead
of the current message log file.

Note: The /flash partition is about 128 MB, and /md partition is 896 MB.
All mandatory logs and data should be backed up in time, otherwise,
the files are overwritten when the disk is full.

Log Files from the Controller Card

Collect system logs from controller card. To collect the logs, enter the save
tech-support log command in exec mode. The logs are stored in the /md
directory on the controller card. You can transfer the file to an external server
using the copy command with the ftp or scp command.

Crash Files from the Controller Card

Verify if related core dump files were generated at the time of the problem
by entering the following commands:

• show crashfiles

• show process crash-info

Transfer the files in binary mode from the router and include an MD5 checksum
of the original file.

Output of the show tech-support Command

You must include the output of the show tech-support command.

Before troubleshooting or performing recovery steps, execute the show

tech-support command in exec mode with no keywords.

The basic command produces output for the following areas:

• Startup and software revision

• System hardware

4 92/1543-AXI 101 09/1 Uen B | 2016-09-28

 Collecting Data Based on Problem Type

• Configuration

• Core system statistics

• Process and memory status and crashes

• Core system processes

• IP routes

• System logs

• Shared memory routing

• Precision Time Protocol (PTP)

If you know that your problem is related to one of the following hardware
or process, you can also run the show tech-support command with an
appropriate keyword to collect the output. See show tech-support for the
keywords.

Network Diagram/Topology

You must provide Network Diagram or Topology including logical protocol level
and actual physical layer 1 port connectivity with IP address of the node in
which issue is reported.

3.2 Acceptable Compression Formats

Note: Ericsson only accepts the following compression formats:

• Compressed tar (*.tar)

• gzip tar (*.tar.gz)

• win zip (*.zip)

4 Collecting Data Based on Problem Type

For more specific problems, collect the output of the following commands.

The show tech-support command can be used with the following optional
keywords to collect troubleshooting data about many Ericsson IP Operating

92/1543-AXI 101 09/1 Uen B | 2016-09-28 5

 Data Collection Guideline

System modules. To collect data for specific problems, use the command in
exec mode with the appropriate keyword.

For example, if you know that your problem is related to BFD, enter the show
tech-support bfd command.

Use the following commands in Table 2 to obtain additional information when a

problem or outage occurs at the customer node. See the appropriate command
reference guide for an explanation of the command syntax.

Note: Because the output of these commands is intended for use by support
engineers, the format might differ from typical show command output
and might not be readable.

Table 2 Types of Monitoring Commands

Type of Command Commands Notes
Monitor a system show chassis Displays status of cards installed in the
component chassis.
show hardware
Displays detailed hardware information.
show card
Displays detailed card information.
show circuit Displays statistics for one or more circuits.
counters
Monitor the status monitor process Monitors the status of a specified category
of a process and of processes, and provides continuous
provide continuous status updates. Enter this command in
updates exec mode.
Monitor files in directory Displays a list of files in the specified
memory directory.
pwd
Displays the current working directory.
Enter these commands in exec mode.
Monitor a process show process Displays status of a process. Enter this
command in any mode.
Display a software show release Displays release and installation
release or version information.
show version
Displays the version of the currently
running OS.
Enter these commands in any mode.

6 92/1543-AXI 101 09/1 Uen B | 2016-09-28

 Severity Levels

Type of Command Commands Notes

Monitor an show privilege Displays the current privilege level for the
administrator current session.
session show public-key
Displays the public keys for an
administrator.
Enter these commands in any mode.
Monitor the system show configuration Displays the configuration commands for a
feature. Enter this command in any mode.
show memory Displays memory statistics. Enter this
command in any mode.
show system alarm Displays system alarms at one or more
levels. Enter this command in any mode.
show port synchrono Displays synchronization information for
us-mode one or all synchronous mode ports.

5 Severity Levels

You must state the severity of the CSR. Applying the correct severity level
ensures that critical problems are fixed before less critical ones. It is important
to follow the guidelines for the four severity types: Emergency, High, Medium,
and Low.

The First-Line Support (FLS) and Second-Line Support (SLS) staff analyze
and determine the severity if the priority is not clear. FLS can increase priority
because of commercial reasons.

5.1 Emergency
Examples of emergencies include:

• Complete system failure—The system does not handle any traffic, and a
manual intervention is needed to restore the system.

• Major disturbance in system functionality—Core capacity of the system is

decreased by more than 30%.

• A billing or charging function stops working or is seriously affected.

92/1543-AXI 101 09/1 Uen B | 2016-09-28 7

 Data Collection Guideline

• Complete loss of I/O, provisioning, or communication for business-critical

systems.

• Critical functionality that affects customer business is not working.

• System startup or reboot fails.

The following problem types are not considered emergencies:

• Consultation requests

• Configuration questions

• Documentation problems

• Change or improvement requests

5.2 High Severity

Examples of high severity problems include:

• Major fault or disturbance that affects a specific area of functionality but

not the whole system.

• Situation that is likely to result in an emergency.

• Major problems or disturbances that require immediate action, such as

large restarts or a failure affecting billing.

• The failure affects connection with other operators.

• The system crashes or hangs.

• Critical functionality is not available.

5.3 Medium Severity

Medium severity problems have minor customer impact. Examples of medium
severity faults include:

• Nonrecurring small or large restarts where the system recovers

automatically

• Questions regarding operation and maintenance

• Documentation errors that cause handling errors

8 92/1543-AXI 101 09/1 Uen B | 2016-09-28

 Severity Levels

5.4 Low Severity

Low severity CSRs must not be related to any fault in the customer network or
network elements. See the SLS for general consultation.

Examples of low severity faults include:

• Questions about network expansion

• Review of documents

• Minor documentation errors

• Configuration consultation

92/1543-AXI 101 09/1 Uen B | 2016-09-28 9

 Glossary

Glossary

AAA TWAMP
Authentication, Authorization, and Accounting Two Way Active Measurement Protocol

BFD
Bidirectional Forwarding Detection

BGP
Border Gateway Protocol

CES
Circuit Emulation Service

CLIs
Command-Line Interfaces

CSR
Customer Service Request

DHCP
Dynamic Host Configuration Protocol

FLS
First-Line Support

IS-IS
Intermediate System-to-Intermediate System

LDP
Label Distribution Protocol

O&M
Operation and Maintenance

OSPF
Open Shortest Path First

PTP
Precision Time Protocol

QoS
Quality Of Service

SLS
Second-Line Support

SNMP
Simple Network Management Protocol

92/1543-AXI 101 09/1 Uen B | 2016-09-28 10