Spectralwave Mn9200: Maintenance Guide

SpectralWave MN9200
Maintenance Guide
System Release 2.5
Doc. Code L2 FL26 2500 10 12 00
NEC Corporation 7-1, Shiba 5-Chome, Minato-Ku, Tokyo 108-8001, Japan

TEL: +81-3-3454-1111
FAX: +81-3-3798-1510
TELEX: NECTOK J22686
Copyright © 2011-2012 NEC. All rights reserved.
Revision History
Manual Version Revision Date Section Affected Change Description
Added more routine maintenance tasks, and added
Routine Maintenance
V1.2 January 4, 2012 Clearing Recycle Bin Manually section
Troubleshooting Added new sections
V1.1 August 13, 2011 All Minor update according to comments
V1.0 June 24, 2011 Routine Maintenance Updated according to comments
V0.2 May 5, 2011 All Updated according to comments
V0.1 March 23, 2011 All Draft
Contents
About This Guide ........................................................................................................1
Introduction ........................................................................................................................1
Conventions.......................................................................................................................1
Notices...........................................................................................................................1
Text................................................................................................................................1
Figures and Screen Captures .......................................................................................2
1 System Requirements .......................................................................................3

Hardware Requirements....................................................................................................3
Software Requirements .....................................................................................................3
Network Requirements ......................................................................................................3
Equipment Room Requirements .......................................................................................3
2 Routine Maintenance .........................................................................................5

Introduction ........................................................................................................................5
Routine Maintenance Tasks ..............................................................................................5
Monitoring CPU, Memory and Disk Space....................................................................5
Daily Maintenance Tasks ..............................................................................................5
Weekly Maintenance Tasks ........................................................................................10
Bi-weekly Maintenance Tasks.....................................................................................13
Monthly Maintenance Tasks........................................................................................13
Server Performance Monitoring Task..............................................................................15
Checking CPU Usage .................................................................................................15
Checking Processor Statistics.....................................................................................15
Checking CPU Usage and Processes’s Memory........................................................15
Checking Virtual Memory Statistics of Process...........................................................16
Checking Terminal and Disk I/O Activity and CPU Utilization.....................................16
Checking JVM Memory Usage....................................................................................16
RDMBS Database Maintenance Task.............................................................................16
Checking Background Processes ...............................................................................16
Checking Database Listener Status ............................................................................17
Checking Database Instance Status ...........................................................................17
Checking Control File and Redo File...........................................................................18
Checking Listing Database File Status........................................................................18
Clearing Recycle Bin Manually....................................................................................18
3 Database Backup and Restoration.................................................................19

RDMBS Database Backup and Restoration (Solaris) .....................................................19
Backing up RDMBS Database ....................................................................................19
SpectralWave MN9200 Maintenance Guide
ii
Restoring RDMBS Database.......................................................................................19

Sybase Database Backup and Restoration (Windows)...................................................19
Backing up Sybase Database .....................................................................................19
Restoring the Database...............................................................................................27
4 MN9200 Upgrading .......................................................................................... 31

Introduction ......................................................................................................................31
Upgrading MN9200 with RDMBS Database....................................................................32
Reinstalling Database Method.....................................................................................32
Upgrading Database Method.......................................................................................33
Upgrading MN9200 with Sybase Database.....................................................................34
Reinstalling Database Method.....................................................................................34
Upgrading Database Method.......................................................................................34
5 Troubleshooting .............................................................................................. 37
Getting File Fails during NE Upgrading ...........................................................................37
NE Lost connection with NMS .........................................................................................37
Unable to Logging in MN9200 .........................................................................................38
PW is Unsynchronized.....................................................................................................40
LSP is Unsynchronized....................................................................................................41
Can’t find Path while creating Multiple segment PW.......................................................41
CE Service is Unsynchronized ........................................................................................42
The MN9200 can’t Restart after changing the IP ............................................................42
Can’t Activate a LSP........................................................................................................45
Illegible Characters in Report when Opening with Excel.................................................46
Unable to Extend Oracle Table .......................................................................................47
Possible Scenario ........................................................................................................47
Possible Cause............................................................................................................47
Solution........................................................................................................................47
New Redo Log cannot be added .....................................................................................50
Possible Cause............................................................................................................50
Impaction on System ...................................................................................................50
Solution........................................................................................................................50
MN9200 Response Unacceptable slow...........................................................................52
Probable Cause ...........................................................................................................52
Solution........................................................................................................................53
MN9200 Server Crashes .................................................................................................56
Maintenance Guide SpectralWave MN9200

iii
A Backup Tool Usage..........................................................................................57

Preparations ....................................................................................................................57
Backing up Database ......................................................................................................57
Restoring Database.........................................................................................................59
Stopping MN9200 Server ............................................................................................59
Restoring MN9200 Database ......................................................................................59

List of Tables
Table 1 Daily Routine Maintenance Tasks..................................................................................5
Table 2 Weekly Routine Maintenance Tasks............................................................................10
Table 3 Monthly Routine Maintenance Tasks ...........................................................................13

ii
List of Figures
Figure 1 View System Performance............................................................................................7
Figure 2 Configure Sybase Servers ..........................................................................................20
Figure 3 Backup Server Name..................................................................................................20
Figure 4 Backup Server Options ...............................................................................................20
Figure 6 Enter Connection ........................................................................................................21
Figure 7 Network Connections..................................................................................................21
Figure 8 New Backup Server Complete....................................................................................22
Figure 9 Configure Sybase Servers ..........................................................................................22
Figure 10 Reinstalling Database Method..................................................................................31
Figure 11 Upgrading Database Method....................................................................................32
Figure 12 Java Temporary Files Setting ...................................................................................39
Figure 13 Java Runtime Environments Settings.......................................................................40

About This Guide
Introduction
The manual describes SpectralWave MN9200 NMS server maintenance.
Conventions
This guide contains notices, figures, screen captures, and certain text conventions.
Notices
The following table lists notice icons used in this guide.
Icon Notice Type Description
Information that contains important features or instructions but is not

Note
hazard-related.
Information to alert of potential damage to a program, data, system, or

device. If not avoided, may result in minor or moderate damage. It may
Caution
also alert against unsafe practices and potential program, data, system,
device damage.
Information to alert of operations that may cause potential accident,
Warning casualty, personal injury, fatality or potential electrical hazard. If not
avoided, could result in death or serious injury.
Information that indicates proper grounding precautions are required
ESD
before handling a product.
Text
The following table lists text conventions in this guide.
Convention Description
This typeface represents text that appears on a terminal
screen, including, configuration file names (only for system
Text represented by output file names), and command names, for example login.
Courier New Font
Commands entered by users are represented by bold, for
example, cd $HOME.
This typeface represents window names, dialog box names,
Text represented by bold tabs, field names, function names, directory, file names, process
names, and commands in text, for example, set the Time field.
Text represented by [Menu] This square brackets represents menus such as [File], and
and [Menu/Sub-menu] [File/New]
Text represented by This angle bracket represents button on screen, function key on
<Button> the keyboard and icon names for example, click <OK>.
Text represented by This typeface represents documents for reference, for example,
Document Name SpectralWave MN9200 Installation Guide

2 About This Guide
Convention Description
Text represented by
This typeface represents files in Unix/Linux system files.
# File format:
Figures and Screen Captures

This guide provides figures and screen captures as examples. These examples
contain sample data. This data may vary from the actual data on an installed system.

System Requirements
1 1
SpectralWave MN9200 NMS system (hereinafter referred to as MN9200)

requirements include hardware, software, network and equipment room requirements.
Hardware Requirements
The hardware required for MN9200 includes host machines for MN9200 servers,
databases and clients. All hardware used in MN9200 system must meet the
requirements specified in the Bill of Materials (BOM).
Software Requirements
The software required for SpectralWave MN9200 includes application software,
database software, client software, LCT software and the third-party database
(Sybase or RDMBS) software.
If third party redundancy solution is provided, for example, by Veritas, the necessary
third party (Veritas) software is required. All software used in MN9200 must meet the
requirements specified in the BOM.
Network Requirements
The MN9200 functions in an IP based Ethernet network environment. A Local Area
Network (LAN) is preferred for better system response time.
An MN9200 network includes networking of servers, databases, printers and
Gateway Network Elements (GNE).
The MN9200 server and database can be installed on one computer. If the MN9200
server and database are installed on separate hosts, a LAN is preferred for the
communication between the server and database. If a Wide Area Network (WAN) is
used for the communication between server and database, the available bandwidth
for the connection between the MN9200 server and the database must be at least
2Mbit/s.
The connections between MN9200 servers and GNEs can be over LAN or WAN. The
bandwidth for each of these connections must be at least 192 kbit/s.
The connection bandwidth between any MN9200 server and client must be at least 2
Mbit/s. Printers and all other hardware used by the MN9200 must be in the same LAN.
If primary and secondary MN9200 servers are deployed, link redundancy must be
provided with bandwidth no less than 2 Mbit/s.
Equipment Room Requirements

The MN9200 equipment room requirements are listed below:
 Centralized power supply

4 Chapter 1 System Requirements
 UPS protection: The UPS must be able to provide power for at least 15 minutes
after a power failure
 Voltage: AC 110V±10V 60Hz or AC 220V±10V 60Hz
 Temperature: 10°C - 30°C
 Humidity: 20% - 80%

Routine Maintenance
2 2
Introduction
Well-planned, effective and reliable maintenance routines significantly reduce
equipment failure rates. It also ensures stable operation and optimum performance of
the equipment.
In addition to minimizing sudden system failures, routine maintenance facilitates the
detection of problem-triggering factors and therefore eliminates potential problems
before they occur. Performing these tasks on regular basis will increase the life span
of equipment and components as well reduce the system-down time.
This chapter focuses on MN9200 routine maintenance procedures.
Routine Maintenance Tasks

The user must execute the following routine maintenance tasks in order to find the
potential risks.
Monitoring CPU, Memory and Disk Space

The user can monitor the CPU, memory and disk space information of one server
through MN9200. The server to be monitored must install system monitor package
first, and then add system monitor tasks in MN9200.
To know more information on how to install and configure the system montor package,
refer to Installing and Configuring System Monitor Package section in SpectralWave
MN9200 Installation Guide (Solaris).
To know more information on how to add system monitor task, refer to System
Monitoring section in SpectralWave MN9200/LCT Operation Guide Volume I.
Daily Maintenance Tasks

Table 1 Daily Routine Maintenance Tasks
Maintenance Item Executed On

Checking Disk Space Every server
Checking Disk Status Every server
Checking CPU Utilization and Memory Every server
Checking Disk IO Status Every server
Checking Archive Files Application server
Checking Equipment Room Environment Every server
Checking Network Problem Every server
Checking Network Environment Every server

6 Chapter 2 Routine Maintenance

Checking NE and MN9200 Host Time Every server
Checking MN9200 server Status Application server
Checking table space usage ratio Database server
Backing up NML Database Database server
Backing up EML Database Database server
Backing up NEL Database Database server
Checking Disk Space (Solaris)

Use the following command to inspect the disk space
#df -k
This command prints the allocation in KB. Capacity, the percentage of normally
available space that is currently allocated to all files on the file system should be
lower than 75%.
Checking Disk Space (Windows)

Use the following procedures to inspect the disk space
1 From the Windows desktop, select [Start>Control Panel]
2 Double click Administrative tools icon in Control Panel
3 Double click Computer Management icon
4 Click Disk Management to view the %Free
The value of %Free should be higher than 15%.
Checking Disk Status (Solaris)

Use the following command to display disk input/output activity. No software errors
(Soft Errors) or hardware errors (Hard Errors) should be displayed.
# iostat -En |grep Error
Checking CPU, Memory and IO Status (Windows)

Use the following procedures to inspect the disk space
1 From the Windows desktop, select [Start>Control Panel]
2 Double click Administrative tools icon in Control Panel
3 Double click Performance icon
4 Click Counter Logs under Performance Logs and Alerts.

Chapter 2 Routine Maintenance 7
Figure 1 View System Performance
5 Double-click System Overview in the right pane to see the properties of System
Overview.
6 Click the begin icon to start the system overview.

7 The result is recorded in C:\PerfLogs\System_Overview.blg.
Note: If user wants to customize the other monitor objects which are not included in
the default System Overview, refer to Microsoft Windows related manual for more
information.
Checking CPU Utilization and Memory

Use the top command to display processor activity, such as memory, CPU state, and
process identification number (PID), in real time.
CPU %idle should be no less than 10%, Memory free should no less than 200M.
#top
Checking Disk IO Status

Use the following command to display the extended device statistics.
#iostat -xcn 5 10
If the disk %b (% time the disk is busy) is always higher than 60%, it means that the
disk IO has problem.
Note: The CPU, memory and disk information can be monitored in GUI through
MN9200 in later version.
Checking Archive Files

By default, the MN9200 server monitors the capacity of the Event Log table,
Operation Log table, System Log table, History NML Alarm table and History EML
Performance table. When the number of records exceeds the predefined threshold, a

TCA will be triggered, and if the number exceeds the maximum value, old data will be
removed or archived automatically.
The monitoring parameters are listed in the below.
 Max Value:
- For EML performance table, when the number of records exceeds this value,
the older data will be removed or archive according to the archive policy.
- For other table, when the number of alarms or logs exceeds the Max Value,
the older alarms, logs or events will be deleted automatically, and the
amount of reserved new alarms and logs is the product of Max Value and
Archive Ratio.
 Rising Threshold (%) When the amount of alarms, logs or events exceeds the
product of Rising Threshold and Max Value, the TCA will be triggered.
 Clear Threshold (%) When the amount of alarms, logs or events is fewer than
the product of Clear Threshold and Max Value, the TCA will be cleaned.
 Reserved Data Ratio The ratio of reserved alarms and logs.
 Archive Directory: The directory where the data is archived. It is the subdirectory
/archive/monitor under the MN9200 application installation directory
Check the configuration and the status of these system monitor tasks
Check whether the archived files meet with the archived condition generated in the
archive directory and check the growing rate of the archived files.
Checking Equipment Room Environment

Power supply, environment temperature and humidity of the equipment room must be
checked daily either by MN9200 users or by qualified professional.
Checking the Network Problems

Use the following command to check the network status:
netstat –i
The number of Collisions (Collis) and Input errors (Ierrs) should be 0. If not, check the
network status.
Checking Network Environment

MN9200 network environment includes the MN9200’s DCN network hardware and
their interconnections. MN9200’s topology checking function can be used to verify if
all NEs are online. Router’s or WS’ connectivity can be verified by using ping and
tracert commands.
The redundant MN9200 server or WS connectivity can be verified only by using ping
and tracert commands.
Pay attention to the response time and packet loss when running the ping command
for connection verification,. Unreliable connection affects the MN9200 system’s
normal functioning.

Checking NE and MN9200 Hosts Time

If the time of an NE and an MN9200 host is not correct, it results in incorrect timing
information in the alarms and performance events. This will lead to misjudgments in
network maintenance and troubleshooting. Therefore, time information must be
accurate. It is recommended to check the time of NEs and MN9200 hosts daily.
Follow the steps below to check the NE time:
1 Select [Maintenance Management>NE Time Management].
2 Select all the NEs, and click <Query NE Time>.
3 If the NE time is not same as server’s time, select the NE and click <Update NE
Time>.
Checking MN9200 Server Status

Follow the steps below to check the MN9200 server status:
1 Log on to the MN9200
2 Check whether the Sync State of each NE in Proxy Management is Synced
3 Check whether the alarm is reported in MN9200 whenever an new alarm is
generated
4 Check whether the performance items are being monitored correctly.
5 Check the latest operation log and record the abnormal operation
Checking Table Space Usage Ratio

Use the following command to check used ratio of the space for each tablespace:
1 Connect SQLPLUS
# su - nmsdb
$sqlplus “/ as sysdba”
2 Execute the following SQL to check table space used ratio:

SQL>select d.tablespace_name,space "sum_space(m)",blocks
sum_blocks,space-nvl(free_space,0) "used_space(m)",
round((1-nvl(free_space,0)/space)*100,2) "used_rate(%)",free_space
"free_space(m)"
from
(select tablespace_name,round(sum(bytes)/(1024*1024),2)
space,sum(blocks) blocks
from dba_data_files
group by tablespace_name) d,
(select tablespace_name,round(sum(bytes)/(1024*1024),2) free_space
from dba_free_space
group by tablespace_name) f
where d.tablespace_name = f.tablespace_name(+)
union all --if have tempfile

select d.tablespace_name,space "sum_space(m)",blocks sum_blocks,

used_space "used_space(m)",round(nvl(used_space,0)/space*100,2)
"used_rate(%)",
nvl(free_space,0) "free_space(m)"
from
(select tablespace_name,round(sum(bytes)/(1024*1024),2)
space,sum(blocks) blocks
from dba_temp_files
group by tablespace_name) d,
(select tablespace_name,round(sum(bytes_used)/(1024*1024),2)
used_space,
round(sum(bytes_free)/(1024*1024),2) free_space
from v$temp_space_header
group by tablespace_name) f
where d.tablespace_name = f.tablespace_name(+);
Backing up NML Database

Refer to Backing up NML Database Only on page 58.
Backing up EML Database

Refer to Backing up EML Database Only on page 58.
Backing up NEL Database

Refer to Backing up NEL Database Only on page 58.
Weekly Maintenance Tasks

Table 2 Weekly Routine Maintenance Tasks

Backing up Historical Alarm Database Database server
Backing up historical performance Database Database server
Backing up event database Database server
Checking Redo log information Database server
Checking Data Files information Database server
Checking Table Information Database server
Checking Index information Database server
Maintaining Client Client PC
Backing up Historical Alarm Database

Refer to Backing up History Alarm Database Only on page 58.

Backing up Historical Performance Database

Refer to Backing up History Performance Database Only on page 58.
Backing up Event Database

Refer to Backing up Event Database Only on page 58.
Checking Redo log Information

Use the following command to check the status of redo log group.
# su -nmsdb
SQL>select * from v$log;
SQL>exit;
Check the status of each redo log group. Only one of the redo log status can be
CURRENT, the other redo log status should be INACTIVE.
If all the redo log status is ACTIVE or CURRENT, it means the switch between redo
log is too frequent, the user should add more redo log groups or resize the redo log.
For more detailed operation, refer to Adding More Redo Log Groups on page 51 .
Checking Data Files Information

Use the following command to check the data files information
# su -nmsdb
SQL>select * from dba_data_files;
SQL>exit;
Check the value of BYTES/ MAXBYTES of each data file, it should be less than 75%.
if any of the value is higher than 75%, you may need to add extend tablespace, refer
to Adding New Data Files to Extend Table Space Size on page 49 for more detailed
operation.
Checking Table Information

Use the following command to check the table information
# su -nmsdb
SQL>select segment_name,sum(bytes)/1024/1024|| 'M ' from
dba_segments where TABLESPACE_NAME= 'NMS_EML_TBLS_DATA' AND
segment_type= 'TABLE' GROUP BY segment_name;
dba_segments where TABLESPACE_NAME= 'NMS_NML_TBLS_DATA' AND
dba_segments where TABLESPACE_NAME= 'NMS_NEL_TBLS_DATA' AND


dba_segments where TABLESPACE_NAME= 'NMS_ALM_TBLS_DATA' AND
dba_segments where TABLESPACE_NAME= 'NMS_EVT_TBLS_DATA' AND
dba_segments where TABLESPACE_NAME= 'NMS_PER_TBLS_DATA' AND
SQL>exit;
Record all the value of each SQL sentence, the sum of these values should be lower
than 75% of the tablespace. If it is higher than 75%, refer to Checking Table Space
Usage Ratio on page 9 to check the tablespace used space.
Checking Index information

Use the following command to check the index table information:
# su -nmsdb
$sqlplus “/ as sysdba
dba_segments where TABLESPACE_NAME= 'NMS_EML_TBLS_IDX' AND
segment_type= 'INDEX' GROUP BY segment_name;
dba_segments where TABLESPACE_NAME= 'NMS_NML_TBLS_IDX' AND
dba_segments where TABLESPACE_NAME= 'NMS_NEL_TBLS_IDX' AND
dba_segments where TABLESPACE_NAME= 'NMS_ALM_TBLS_IDX' AND
dba_segments where TABLESPACE_NAME= 'NMS_EVT_TBLS_IDX' AND
dba_segments where TABLESPACE_NAME= 'NMS_PER_TBLS_IDX' AND
SQL>exit;
Record all the value of each SQL sentence, the sum of these values should be lower
than 75% of the tablespace. If it is higher than 575%, refer to Checking Table Space
Usage Ratio on page 9 to check the tablespace used space.
Maintaining Client
MN9200 client software is a JAVA program running on Windows XP Professional
operating system. Since the response time of a JAVA program degrades after
running for a long period of time, the MN9200 client software must be restarted at
least once every month.
Follow the steps below to restart MN9200 client:

 Exit all MN9200 client applications
 Restart the computer
 Run the MN9200 client applications
After a prolonged period of continuous operation, computer’s hard disks are badly
fragmented and the computer’s performance is significantly degrades. Therefore,
monthly defragmentation must be performed.
Defragmentation consumes significant CPU and disk space. It is recommended to
exit all other applications before performing defragmentation.
Note: All data related to alarms and configuration is stored in the database. Therefore
shutdown and restart of the WS does not result in any data loss.
Bi-weekly Maintenance Tasks

De-fragmenting Oracle Table Space
De-fragment table space by executing command shrink table to degrade table’s HWM
(high water mark) every two weeks.
For example, defragment the performance database.
1 Query all tables belong to tablespace MN9200 performance database
SQL> select table_name from dba_tables where owner='NMS_PER';
2 Drop the tables whose name is like ‘1970xxxx’, ‘2035xxxx’ etc and whose date is
not current data.
3 Query all tables belong to tablespace MN9200 performance database again
4 Using the tables name to edit below two SQLs

SQL>alter table table_name enable row movement;
SQL>alter table table_name shrink space cascade;
5 Execute the two set of SQLs orderly, firstly alter table table_name enable row
movement; then alter table table_name shrink space cascade;
Because the table size is big, it will need long time to execute the SQLs, please wait
patiently.
Monthly Maintenance Tasks

Table 3 Monthly Routine Maintenance Tasks

Maintaining MN9200 server Application server
Changing the NMS user password Application server
Backing up NE Backup Configuration Data Files Application server
Archiving Performance Data Application server


Backing up Report Files Application server
Backing up archive files Application server r
Backup Oracle alert log Database server
Backup Oracle trace log Database server
Maintaining the MN9200 Server

1 Check all hardware connections, including the power supply cable, Ethernet
cables and other cables.
2 Use a dedicated socket for the power supply. Keep the cabling neat and tidy.
Caution: Do not use the NMS server for any other purpose. It is best to isolate the
NMS computer from external networks, especially the Internet.
Changing the NMS Administrator Password

Keep the password of all the NMS users confidential, and change it once every three
months.
Follow the steps below to change the password:
1 Select [Security>User Management].
2 Select the user and click <Change Password>.
Backing up NE Backup Configuration Data Files

If NE configuration backup tasks are created, then the NEs will be backed up at the
specific time in the /neconfs directory under the MN9200 application installation
directory.
Check the Status and the Last Run Result of the tasks.
Check the number of the files and backup the NE configuration data files under
/neconfs monthly.
Archiving Performance Data

By default, the 15 minutes performance data file are generated every 15 minutes, and
24 hours performance data of is generated every 24 hours for every NE after the
MN9200 server startup.
Move or delete the performance data files in the subdirectory /pers under the
MN9200 application installation directory to other media such as another disk or type
monthly.
Backing up Report Files

Check the report template and backup the report generated under the /ureport
directory under the MN9200 application installation directory.

Backing up Archive files

Backup the archive files under the directory $MN9200_installation_folder/archive
on the application server.
Backing up Oracle Log Files

Backup the Oracle alert log under the /opt/nmsdb/admin/ms/bdump/nms_alert.log
Backup the Oracle trace log files *.trc under /opt/nmsdb/admin/nms/bdump and
/opt/nmsdb/admin/nms/udump on database server.
Backing up Oracle Trace Files

Backup all the *.trc files under /opt/nmsdb/admin/nms/bdump, and *.trc files under
the /opt/nmsdb/admin/nms/udump/ on database server.
Server Performance Monitoring Task

The use can use the following tasks to collect the server performance information. It
is suggested to perform the tasks often.
Checking CPU Usage

Use the following command to show CPU usage over at one minute interval in
2000000 reports.
sar 60 2000000 >> filename
Where, filename is the file which the information stored in.

The result of average (%idle) should be higher than 40% always. If not, contact the
field engineer to check out this problem.
Checking Processor Statistics

Use the following command to collect the processor statistics in tabular form one
minute interval.
mpstat 60 >> filename
The average (Idl) of each processor should be higher than 40% always. If not, contact
the field engineer to check out this problem.
Checking CPU Usage and Processes’s Memory

Use the following command to report information about major processes's memory
and CPU usage at one minute interval.
prstat –a 60 >> filename
The main java process's SIZE should be lower than 60% of the physical memory, and
the value of RSS should be lower than that of Size. The main java process's CPU
usage on application server should be lower than 60%. If not, contact the field
engineer to check out this problem.

Checking Virtual Memory Statistics of Process

Use the following command to collect virtual memory statistics of process, virtual
memory, disk, trap, and CPU activity memory (in Kbytes) at one minute interval.
vmstat 60 >> filename
The free (KB) should be higher than or equal 20% of the physical memory always. If
not, contact the field engineer to check out this problem.
Checking Terminal and Disk I/O Activity and CPU Utilization

Use the following command to collect terminal and disk I/O activity and CPU
utilization at one minute interval.
iostat -xtcn 60 >> filename
The value of averavge(%b) should be less than or equal to 80%.
Checking JVM Memory Usage

Use this command to collect the JVM memory usage of the application server every
minute.
/usr/jdk/instances/jdk1.6.0/bin/jstat -gc PID 60000 0 >> filename
The sum of EC+OC+PC should be less than or equal to the maximum JVM value,
and the sum of EU+OU+OU should be lower than the value of the (maximum JVM
value -200M ).
If not, contact the field engineer to check out this problem.
RDMBS Database Maintenance Task

The user can perform the following task to check the RDMBS database:
 Checking Background Processes
 Checking Database Listener Status
 Checking Database Instance Status
 Checking Control File and Redo File
 Checking Listing Database File Status
Checking Background Processes

The following background processes that start when an Instance is started.
 Database Write (DBWn) is the process that writes data files.
 Log Writer (LGWR) is the process that writes log files.
 System Monitor (SMON) is the process that monitors the instance status.
 Process Monitor (PMNO) is the process that monitors the client connection to the
database.
 Checkpoint (CKPT) is the process that updates the headers of all data files to
record the checkpoint details.

Use the following command to display the background processes:

# ps –ef|grep ora
Expected Result
Any of the following background processes of Oracle: dbw0, lgwr, pmon, smon,
ckpt should be listed. The name behind the process name is the instance name of
the Oracle database which created during Oracle installation.
Example Output
Checking Database Listener Status

Use the following command to display the connection to the RDMBS Database.
# lsnrctl status
Service Summary lists the database instance the Oracle listener is listening .The
instance name nms in Services Summary should be listed.
Checking Database Instance Status

Use the following commands to display the instance to which the user is currently
connected and its status.
The status list the status of current instance, the status of the instance nms should be
OPEN.
# su -nmsdb
SQL> select instance_name,status from v$instance;
Example Output

Checking Control File and Redo File

 Check control files
Use the following command to display the associated control files of the Oracle
instance.
STATUS should be null and there should be at least three entries under NAME.
SQL> select * from v$controlfile;
 Check Redo Log file

Use the following command to display the associated log files of the database
instance. The status of three redo logs should be ONLINE.
SQL> select * from v$logfile;
Checking Listing Database File Status

Use the following command to display the location of each data files, status, and size
in megabytes.
SQL> select name, status, enabled, bytes/1024/1024 sizeMB from
v$datafile;
Clearing Recycle Bin Manually

If the database is Oracle 10g, use the following command to clear the recycle bin
manually if necessary:
SQL> purge recyclebin;
Recyclebin purged.
SQL> select * from recyclebin;
no rows selected

Database Backup and Restoration
3 3
This section describes how to back up and restore MN9200 database.

If MN9200 uses RDMBS database, refer to RDMBS Database Backup and
Restoration (Solaris) , and refer to Sybase Database Backup and Restoration
(Windows) if Sybase database is used.
RDMBS Database Backup and Restoration (Solaris)

Backing up RDMBS Database
For detailed operational procedures, refer to section Backing up Database on page
57 .
Restoring RDMBS Database

For detailed operational procedures, refer to section Restoring Database on page 59.
Sybase Database Backup and Restoration (Windows)

This section describes how to backup and restore the NMS database.
The NMS database includes the following six databases:
 NML Database
 EML Database
 NE Database
 History Alarm Database
 History Performance Database
 Event Database
The database name for each database is defined while installing MN9200 database.
Backing up Sybase Database

The Sybase backup server must be created first for database backup, and then the
database and transaction log can be dumped to other devices.
Creating the Backup Server

1 From the Windows Start menu, select [Start>Programs>Sybase>Adaptive Server
Enterprise/Server config].
2 The Configure Sybase Servers window appears.
3 Click Backup Server in the Products List.
4 Click <Create Backup Server…>.

20 Chapter 3 Database Backup and Restoration
Figure 2 Configure Sybase Servers
5 Enter the backup server name, such as nms_BS.

Click <Continue>.
Figure 3 Backup Server Name
6 In the Backup Server Options window, click <Network Addresses…>.

Figure 4 Backup Server Options
7 In the Network Connections window, click <Add>.

Chapter 3 Database Backup and Restoration 21
Figure 5 Network Connections
8 In the Connection Info text box, type the backup server’s hostname (or IP
address) and port number (4200) separated by a comma, for example:
172.21.65.49,4200.
Click <OK>.
Figure 6 Enter Connection
9 In the Network Connections window, click <OK>.

Figure 7 Network Connections
10 Configuring the new backup server is complete.

Click <Continue>.

Figure 8 New Backup Server Complete
11 In the Configure Sybase Servers window, click <Exit>.

Figure 9 Configure Sybase Servers
Checking the Backup Service

Check whether the backup service is installed.
1 Open the Command Prompt window.
2 Connect to the Sybase database:
isql -Usa -P –Snms
3 Execute the following commands to check the backup service:

select srvname,srvnetname from master..sysservers where
srvname='SYB_BACKUP'
go
The following output displays that srvnetname is nms_BS, indicating the

Sybase backup server is installed correctly.
srvname srvnetname
-------------- ---------------
SYB_BACKUP nms_BS

4 If the srvnetname is null, you need to execute the following commands to update
it to nms_BS:
sp_configure "allow update", 1
go
use master
go
update sysservers set srvnetname="nms_BS" where
srvname="SYB_BACKUP"
go
select srvname, srvnetname from master..sysservers where
srvname='SYB_BACKUP'
go
Checking the Database Consistency

Before backing up the database, use the dbcc commands to check its consistency,
because the dump commands can be executed successfully even if the database is
corrupted.
Creating the Dbcc Database
3 Run sp_plan_dbccdb in the master database to obtain recommendations for the

database size, devices, workspace sizes, cache size, and the number of
processes for the target database.
To restore several databases, repeat the following steps for each database.
For example, suppose you run sp_plan_dbccdb with history alarm database
nms_alm as the target database when dbccdb does not exist.
Execute the following commands:
use master
go
sp_plan_dbccdb nms_alm
go
The following is output:

Recommended size for dbccdb database is 23MB (data=21MB, log=2MB).
Recommended devices for dbccdb are:
Logical Device Name Device Size (KB)
----------------------- -------------------------
syb_tempdb 4096000
Recommended values for workspace size, cache size and process
count are:
dbname scan ws text ws cache process count

nms_alm 1856K 512K 1920K 3
4 Create a data cache for dbccdb to improve the checking performance. The
cache size must be larger than the result scanned by sp_plan_dbccdb.
For example, to create a 12M cache, execute the following commands:
sp_cacheconfig nms_alm_cache, '12M'
go
5 If dbccdb already exists, drop it and all associated devices before creating a new
dbccdb database:
use master
go
if exists (select * from master.dbo.sysdatabases
where name = "dbccdb")
begin
print "+++ Dropping the dbccdb database"
drop database dbccdb
end
go
6 Use disk init to initialize disk devices for the dbccdb data and the log. The
device size must be no less than the value scanned by sp_plan_dbccdb.
use master
go
disk init
name ='dbccdb_dat',
physname = 'd:\sybase15\data\dbccdb.dat',
size ='21M'
go
disk init
name = 'dbccdb_log',
physname = 'd:\sybase15\data\dbcclog.dat',
size = '2M'
go
7 Use create database to create dbccdb on the data disk device created in the
previous step. The database size must be no less than the value scanned by
sp_plan_dbccdb.
use master
go
create database dbccdb

on dbccdb_dat = 21
log on dbccdb_log = 2
go
8 Add segments for the scan and text workspaces to the dbccdb data device.
use dbccdb
go
sp_addsegment scanseg,dbccdb,dbccdb_dat
go
sp_addsegment textseg,dbccdb,dbccdb_dat
go
9 Create tables for dbccdb.

isql -Usa -P -Snms -id:\sybase15\ASE-15_0\scripts\installdbccdb
The installdbccdb script checks for the existence of the database before it
attempts to create the tables. It only creates the tables that do not exist in
dbccdb. If any of the dbccdb tables is corrupted, drop it and then use
installdbccdb to re-create it.
10 Create and initialize the scan and text workspaces. The workspace size must be
no less than the value scanned by sp_plan_dbccdb.
use dbccdb
go
sp_dbcc_createws dbccdb, scanseg, scan_aca,scan,"35008K"
go
sp_dbcc_createws dbccdb, textseg,text_aca,text,"8768K"
go
11 Set the number of worker processes as the value scanned by sp_plan_dbccdb.

sp_configure "number of worker processes", 9
go
12 Update the dbcc_config table.

use dbccdb
go
sp_dbcc_updateconfig nms_alm,"max worker processes","2"
go
sp_dbcc_updateconfig nms_alm,"dbcc named
cache",nms_alm_cache,"12M"
go
sp_dbcc_updateconfig nms_alm,"scan workspace",scan_aca
go
sp_dbcc_updateconfig nms_alm,"text workspace",text_aca

go
sp_dbcc_updateconfig nms_alm,"OAM count threshold","5"
go
sp_dbcc_updateconfig nms_alm,"IO error abort","3"
go
sp_dbcc_updateconfig nms_alm,"linkage error abort","8"
go
Checking the Database Consistency

15 To prevent concurrent updates, set the database in single user mode.

sp_dboption nms_alm,"single user", true
go
16 Check the page linkage and data pointers at both the page level and the row
level using checkdb.
use dbccdb
go
dbcc checkdb(nms_alm)
go
17 Check the page allocation using checkalloc.

use dbccdb
go
dbcc checkalloc(nms_alm)
go
18 Use checkcatalog to check for consistency within and between the system
tables in a database.
use dbccdb
go
dbcc checkcatalog(nms_alm)
go
Dumping the Database

If the backup server is not running, start the Sybase service manually.
3 Make the database run in single user mode with the admin user sa.
sp_dboption nms_alm,"single user", true

go
4 The command format database dumping is:

dump database database_name to file_name
In this section, take history alarm database as an example.

The history alarm database name is defined during the MN9200 server
installation, it is nms_alm by default.
Suppose backup file to be stored in “d:\sybase15\nmsalm_backup.dmp”.
Execute the following commands.
dump database nms_alm to "d:\sybase15\nmsalm_backup.dmp"
go
A database can be dumped to several files, for example:

dump database nms_alm to " d:\sybase15\backup.dmp.1"
stripe on "d:\sybase15\backup.dmp.2"
go
When dumping a database, you can use the compress option to compress files:
dump database nms_alm to "compress:: d:\sybase15\backup.dmp.1"
stripe on "compress:: d:\sybase15\backup.dmp.2"
go
5 Finally, do not run the database in single user mode with the admin user sa.
sp_dboption nms_alm,"single user", false
go
Restoring the Database

The restore command format is:
load database database_name from file_name

3 For example, to restore the history alarm database nms_alm using the backup
file, execute the following commands:
load database nms_alm from " d:\sybase15\backup\nmsalm_backup.dmp"
go
online database nms_alm
go

To load the database from several files, execute the following commands:
load database nms_alm f from "d:\sybase15\backup.dmp.1"
go
go
To load the database from the compressed files, execute the following
commands:
load database nms_alm from "compress::d:\sybase15\backup.dmp.1"
go
go

MN9200 Upgrading
4 4
MN9200 upgrade must be performed by system administrators.

If there is a secondary MN9200 server, the primary MN9200 server must be
upgraded first.
Introduction
The administrator can upgrade the MN9200 in either of the following methods:
Reinstalling Database Method
In this method, the user backs up the old database and then restores the old
database in the new MN9200 database.
Figure 10 Reinstalling Database Method
Upgrading Database Method

In this method, the user reinstalls the MN9200 application without reinstalling
database.

32 Chapter 4 MN9200 Upgrading
Figure 11 Upgrading Database Method
Upgrading MN9200 with RDMBS Database

In this method, upgrade MN9200 includes the following steps:
1 Stop MN9200 application.
Refer to section Stop MN9200 application on page 32.
2 Back up old database
Refer to section Backing up Database on page 57 to backup the old MN9200
database.
3 Uninstall Old Database
Refer to section Uninstall Old Database on page 33.
4 Install new MN9200 Database software in a different directory
5 Restore backed up Data to new database
Refer to section Restoring Database on page 59.
6 Install new MN9200 application software, activate the license file
7 Startup new MN9200
Refer to SpectralWave MN9200 Installation Guide (Solaris) for more details
8 Log in MN9200 and check the data is same as the data before backing up
Stop MN9200 application

1 Enter the MN9200 application server as root.
Chapter 4 MN9200 Upgrading 33
2 Enter the subdirectory /bin under the MN9200 application installation directory
3 Execute the following command
#./stop_all.sh
Uninstall Old Database

1 Log on to the MN9200 server as nmsdb.
2 Enter the old MN9200 data base directory, and execute the script to uninstall the
old database
$./uninstall_all.sh

1 Stop MN9200 application
2 Install new MN9200 application
Refer to section Install new MN9200 application on page 33 .
3 Copy Database Data
Refer to section Copy Database data on page 33.
4 Start new MN9200 server
Refer to section Start new MN9200 Server on page 33.
5 Check the integrity of the data
Refer to section Check the integrity of the data on page 34 .
Install new MN9200 application

2 Install new MN9200 application software in a different directory
3 Activate the applied license file
Copy Database data

2 Enter the following files in subdirectory /usercfg under the old MN9200
application installation directory
3 Copy the files usingdb.lua db_oracle.lua to the subdirectory /usercfg under the
new MN9200 application installation directory
# cd $MN9200_appoldversion/usercfg
# copy usingdb.lua db_oracle.lua $MN9200_appnewversion/usercfg
Start new MN9200 Server

Refer to SpectralWave MN9200 Installation Guide (Solaris) for more details

34 Chapter 4 MN9200 Upgrading
Check the integrity of the data

1 Log in MN9200 system
2 Check whether the number of NEs and the configuration of NEs are same with
the previous MN9200 system
Upgrading MN9200 with Sybase Database

1 Stop MN9200 application
2 Back up old database
Refer to Backing up Sybase Database on page 19 to backup the old MN9200
Sybase database.
3 Uninstall the old MN9200 Database
4 Install the new MN9200 Database software
5 Restore the old database data in the new database
Refer to Restoring the Database on page 27.
6 Install new MN9200 application software, activate the license file
7 Start new MN9200 server
Refer to Start new MN9200 Server on page 35.
8 Log in MN9200 and check the data is same as the data before backing up

Stop MN9200 application

From the Windows desktop, select
[Start>Program>SpectralWave_MN9200_2.5>Shutdown Server] and shutdown the
MN9200.
Install new MN9200 application

1 Install new MN9200 application software
2 After installing the MN9200 server, the user must copy the License key (The file
name is license.xml.sig ) to the MN9200 server installation folder.
Copy Database data

1 Enter the following files in subdirectory /usercfg under the old MN9200
application installation directory
2 Copy the files usingdb.lua db_sybase_.lua to the subdirectory /usercfg under
the new MN9200 application installation directory
Chapter 4 MN9200 Upgrading 35
Start new MN9200 Server

From the Windows desktop, select
[Start>Program>SpectralWave_MN9200_2.5>Start Server] and start the MN9200
Server applications.
Check the integrity of the data

1 Log in MN9200 system
2 Check whether the number of NEs and the configuration of NEs are same with
the previous MN9200 system

Troubleshooting
5 5
This chapter describes the process and methods of troubleshooting MN9200 faults.
Getting File Fails during NE Upgrading

Possible Scenario
After the user begins upgrading operation, the process of getting files fails.
Probable Cause
1 The FTP setting is wrong
2 The NE upgrade files are not correct
3 The flash of the NE is wrong.
Solution
 The FTP setting is wrong
1 Ensure that upgrade files are copied to the specified directory of FTP server.
2 Check the FTP setting: Host is the FTP server’s IP address and Path is the
directory where the upgrade files locates.
3 Ensure that the FTP user which username is weihu and password is cjhyy300
exists in the FTP server
 The NE upgrade files are not correct
1 Check the NE upgrade files, and ensure the upgrade files are transferred to FTP
server in Bin Mode.
 The flash of the NE is wrong
1 Check whether the flash of the NE is full. If it is full, remove some old files or
format the flash.
2 If the flash doesn’t exist on NE, format the NE
NE Lost connection with NMS

Possible Scenario
NE lost the connection with MN9200 and the user can not do any operation on this
NE.
Probable Cause
1 Network connection fails
2 The proxy server fails to work
3 The NE is out of power
4 The NE version and the MN9200 version doesn’t match
38 Chapter 5 Troubleshooting
5 The configuration of NE is not correct
Solution
 Network connection fails
Check the Ethernet network connection between the NE and MN9200.
 The configuration of proxy is not correct.
Check the proxy configuration
- If the master proxy is not the default proxy, check the status of the proxy.
And check the alarm with Alarm_ID Proxy_Connection_Lost in Current
EML Alarm window
- If the master proxy is the default proxy, ping the NE’s IP address on MN9200
server. If the ping operation is successful, connect with NE directly to check
the configuration of NE
- Check the NE status
 The NE is out of power
Check the power of the NE and start the NE.
 The NE version and the MN9200 version doesn’t match
Check the MN9200 version and the NE version.
 The configuration of NE is not correct
Check the interface status, OSPF configuration and other NE configuration.
Unable to Logging in MN9200

Possible Scenario
1 None of the clients can log on to the NMS System
2 Some clients can’t log on to the NMS System.
Probable Cause
1 The version of the client and server mismatches.
2 The MN9200 server disk or Database server disk is full.
3 Java configuration is incorrect.
4 JRE version is not correct
5 The client Internet Explorer configuration is incorrect.
Solution
 The version of the client and server mismatches.
Reinstall the version of the client and ensure the version of the client and server
is the same.
 The MN9200 server disk or Database server disk is full.
1 Log on to the MN9200 server or database server.
Chapter 5 Troubleshooting 39
2 Check that there is sufficient disk space.

#df –k
If the allocated space is more than 75%, please clear the unused files and free more
space
 JRE Configuration is not correct
1 Determine if JRE has been installed correctly in the client. Ensure the Java Web
Start configuration is correct.
a Click start, and click Control Panel
b Double-click Java.
c Click General tab.
d Click <Settings…> in Temporary Internet Files frame.
e Click <Delete Files…>.
f Click <OK>.
Figure 12 Java Temporary Files Setting
2 If the user still can’t log in MN9200, then reinstall the JRE.
 JRE version is not correct
Check the JRE version, if it is not higher than 1.6, reinstall the JRE.
Follow the steps below the check the version of JRE:
a Click start, and click Control Panel
b Double-click Java.
c Click Java tab, and click <View…>

Figure 13 Java Runtime Environments Settings
 The client Internet Explorer configuration is incorrect.

1 Launch the Internet Explorer;
2 On the Tools menu, click the Internet Options, and then click the Connections
tab.
3 Click <LAN Settings>.
4 In the Automatic Settings group, clear Automatically detect settings.
5 In the Proxy server group, clear Use a proxy server
6 Click <OK>.
PW is Unsynchronized
Possible Scenario
The Sync Status of a PW is Unsync.
Probable Cause
The PW configuration on NE and MN9200 are inconsistence.
Solution
 The PW configuration on NE and MN9200 are inconsistence.
1 Click the sync check icon to check the PW label and Qos configuration on
NE and the MN9200
2 Update the data on MN9200 according to the result of step 1.
3 Click the sync check icon again to check.

4 Deactivate the LSP and reactivate it.

LSP is Unsynchronized
Possible Scenario
The Sync Status of a LSP is Unsync.
Probable Cause
The LSP configuration on NE and MN9200 are inconsistence.
Solution
 The LSP configuration on NE and MN9200 are inconsistence.
1 Click the sync check icon to check the configuration on NE and the MN9200
3 Click the sync check icon again to check.
4 Click the compare icon to compare LSP ID, Reversed LSP ID, Tunnel ID,
APS ID or Reversed APS ID on NE and MN9200.
5 To synchronize the data on MN9200 to that of NE, select the NE in the list, and
click <OK>.
6 Deactivate the LSP and reactivate it.
Can’t find Path while creating Multiple segment PW

Possible Scenario
When the user creates multiple segment PW, if selects Create Route Automatically
to create the route, the expected route can’t be found.
Probable Cause
The path is too long and the hops are more than the value of Allowed Maximum Hops.
Solution
Set the value of parameter Allowed Maximum Hops higher than the hops that the
path crosses

CE Service is Unsynchronized
Possible Scenario
The Sync Status of CE is Unsync..
Probable Cause
1 The CE configuration on NE and MN9200 are inconsistence.
Solution
 The configuration on NE and MN9200 are inconsistence.
1 Click the icon to check the data such as AC numbers, properties of AC and
Qos configuration on NE and the MN9200
The MN9200 can’t Restart after changing the IP

Possible Scenario
The MN9200 server (in Solaris system) can’t start successfully after changing the IP
address of the database server.

Probable Cause
1 The database server is shutdown
2 The MN9200 application can not connect to database
Solution
 The database server is shutdown
Restart the database server
 The MN9200 application can not connect to database
After IP change, it is not necessary to reinstall server, the following three
configuration files should be modified:
1 Check the file $ORACLE_HOME/network/admin/tnsnames.ora, and modify the
value of the HOST to the actual IP address. If the value of the HOST is
server_hostname, do not to change it.
NMS =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.4.120.120)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = nms)
)
)
2 Check the file $ORACLE_HOME/network/admin/listener.ora , and modify the

value of the HOST to the actual IP address. If the value of the HOST is
server_hostname, do not to change it.
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.4.120.120)(PORT = 1521))
)
)
3 Check the file $MN9200_installation_folder/usercfg/db_oracle.lua, and

modify the value of hostname to the actual IP address
-- storage nml data
local nmldb = {
username="nms_nml255",
password=" nms_nml255",
hostname="10.4.120.120",
databaseport=1521,
databasename="nms";
type="oracle"
}
-- storage eml data

local emldb = {
username="nms_eml2556",
password="nms_eml255",
hostname="10.4.120.120",
databaseport=1521,
databasename="nms";
type="oracle"
}
-- storage ne cache data

local neldb = {
username="nms_nel255",
password="nms_nel255",
hostname="10.4.120.120",
databaseport=1521,
databasename="nms";
type="oracle"
}
-- storage history performance data
local perdb = {
username="nms_per255",
password="nms_per255",
hostname="10.4.120.120",
databaseport=1521,
databasename="nms";
type="oracle"
}
-- storage history alarm data
local almdb = {
username="nms_alm255",
password="nms_alm255",

hostname="10.4.120.120",
databaseport=1521,
databasename="nms";
type="oracle"
}
-- storage event and operation data
local evtdb = {
username="nms_evt255",
password="nms_evt255",
hostname="10.4.120.120",
databaseport=1521,
databasename="nms";
type="oracle"
}
local _dbcfg={
nmlcfg = nmldb,
emlcfg = emldb,
secu = emldb,
inv = emldb,
nelcfg = neldb,
per = perdb,
emlalm = almdb,
nmlalm = almdb,
evtdb = evtdb
The first two files maybe need not to be changed, but if changed, please restart
Oracle instance.
The last file must be changed, restart the MN9200 application after changing.
Can’t Activate a LSP

Possible Scenario
The user can’t activate one a planned LSP.
Probable Cause
Dummy data was found on MPLS Protection and General MPLS of the related NE.

Solution
 Dummy data was found on MPLS Protection and General MPLS of the related
NE
Delete the dummy data in the following order
- In the related DVM management, delete corresponding APS ID in MPLS
Protection
- In the related DVM management, delete the corresponding LspID in MPLS
OAM tab of General MPLS
- In the related DVM management ,delete the corresponding Tunnel Name of
Tunnel Config tab of General MPLS
- In the related DVM management , delete the corresponding Mplsid in MPLS
Policy Map tab of QoS Management -
- In the related DVM management , delete corresponding Lsp Id in Mpls XC
Config tab of General MPLS
Illegible Characters in Report when Opening with Excel

Possible Scenario
The inventory report can be exported as CSV file. If this report contains Chinese or
Japanese characters, illegible characters are displayed when the file is opened with
Excel.
Probable Cause
The encoding of the CSV file is not correct. When the user opens a CSV file use
Excel, the Excel take granted that the encoding of the file is ANSI. If the encoding of
the CSV file is not ANSI, illegible characters are displayed.
Solution
Modify the encoding type of the CSV file to ANSI.
1 Open the CSV file with Notepad
2 Select [File>Save As]. In the Save As dialog box, ensue the .csv is the suffix of
the file. The ANSI should be selected in Encoding dropdown list.
3 Open the new CSV file using Excel. No illegible characters displayed.

Unable to Extend Oracle Table

Possible Scenario
Oracle alert log (nms_alert.log under subdirectory /admin/nms/bdump of Oracle
installation) including the following error information:
ORA-1653: unable to extend table
ORA-1654: unable to extend index
Fri Jul 22 12:24:57 2011
ORA-1653: unable to extend table NMS_PER.T_PER15_20110722 by 128 in
tablespace NMS_PER_TBLS_DATA
ORA-1653: unable to extend table NMS_PER.T_PER15_20110722 by 1024 in
Sat Jul 23 09:37:01 2011
ORA-1654: unable to extend index NMS_PER.PK_PER15_20110723 by 128 in
ORA-1654: unable to extend index NMS_PER.PK_PER15_20110723 by 1024 in
Possible Cause
The table space has many fragments, which lead to Oracle fails to allocate an extent
of the required number of blocks for a table segment in the table space indicated.
This error does not necessarily indicate whether or not you have enough space, it
merely indicates that Oracle could not find a large enough area of free contiguous
space in which to fit the next extent.
Solution
Querying Tablespace Used Ratio
1 Connect SQLPLUS
# su - nmsdb
2 Execute the following SQL to check tablespace used ratio,

SQL> SELECT D.TABLESPACE_NAME,SPACE "SUM_SPACE(M)",BLOCKS
SUM_BLOCKS,SPACE-NVL(FREE_SPACE,0) "USED_SPACE(M)",
ROUND((1-NVL(FREE_SPACE,0)/SPACE)*100,2) "USED_RATE(%)",FREE_SPACE
"FREE_SPACE(M)"
FROM
(SELECT TABLESPACE_NAME,ROUND(SUM(BYTES)/(1024*1024),2)
SPACE,SUM(BLOCKS) BLOCKS
FROM DBA_DATA_FILES
GROUP BY TABLESPACE_NAME) D,
(SELECT TABLESPACE_NAME,ROUND(SUM(BYTES)/(1024*1024),2) FREE_SPACE
FROM DBA_FREE_SPACE
GROUP BY TABLESPACE_NAME) F
WHERE D.TABLESPACE_NAME = F.TABLESPACE_NAME(+)
UNION ALL --if have tempfile
SELECT D.TABLESPACE_NAME,SPACE "SUM_SPACE(M)",BLOCKS SUM_BLOCKS,
USED_SPACE "USED_SPACE(M)",ROUND(NVL(USED_SPACE,0)/SPACE*100,2)
"USED_RATE(%)",
NVL(FREE_SPACE,0) "FREE_SPACE(M)"
FROM
(SELECT TABLESPACE_NAME,ROUND(SUM(BYTES)/(1024*1024),2)
SPACE,SUM(BLOCKS) BLOCKS
FROM DBA_TEMP_FILES
GROUP BY TABLESPACE_NAME) D,
(SELECT TABLESPACE_NAME,ROUND(SUM(BYTES_USED)/(1024*1024),2)
USED_SPACE,
ROUND(SUM(BYTES_FREE)/(1024*1024),2) FREE_SPACE
FROM V$TEMP_SPACE_HEADER
GROUP BY TABLESPACE_NAME) F
WHERE D.TABLESPACE_NAME = F.TABLESPACE_NAME(+);
Check the used ratio of data and index for each database , especially
NMS_PER_TBLS_DATA and NMS_PER_TBLS_IDX. If the used ratio of the table
space is higher than 75%, de-fragment Oracle Tablespace.
De-fragmenting Oracle Table Space
De-fragment table space by executing command shrink table to degrade table’s HWM
(high water mark) every two weeks.
1 Query all tables belong to tablespace NMS_PER_TBLS_DATA
2 Drop the tables whose name is like ‘1970xxxx’, ‘2035xxxx’, the date is not the
current data.
3 Edit the SQL sentence shrink.sql in another window
$vi shrink.sql
Write the following sentence in the shrink.sql

SET ECHO OFF;
SET FEEDBACK OFF;
SET VERIFY OFF;
SET PAGESIZE 0;
SET TERMOUT ON;
SET HEADING OFF;
ACCEPT username CHAR PROMPT 'Enter the username: ';
spool shrinklistperformance.sql;

SELECT
'ALTER TABLE ' || '&username' || '.' || table_name || ' enable row
movement;'
FROM dba_tables where owner='&username';
SELECT
'ALTER TABLE ' || '&username' || '.' || table_name || ' shrink
space cascade;'
FROM dba_tables where owner='&username';
exit
4 Execute the script shirink.sql

$sqlplus system/manager
SQL>@shrink.sql
5 Enter the username of the database. For example, if the user name of
performance database is nms_per. Enter the nms_per
Enter the username:nms_per
6 Execute the sql name specified in shirink.sql In this example, it is the

shrinklistperformance.sql.
@shrinklistperformance.sql
7 To de-fragment tables for other database, change the shrinklist.sql to other

name in shrink.sql script (shrinkalarm.sql for example), and repeat step 4 to step
6.
Because the table size is big, it will need long time to execute the SQLs, please wait
patiently.
After the de-fragment operation completes, query the tablespace used ratio once
again. If the used ratio tablespace is still higher than 75%, add new data files to
extend table space size.
Adding New Data Files to Extend Table Space Size
1 Query the current data files of NMS_PER_TBLS_DATA
SQL> select file_name from dba_data_files where
tablespace_name=’NMS_PER_TBLS_DATA’;
The example output is:

FILE_NAME
------------------------------------------------------------------
--------------
/opt/nmsdb/oradata/data/nms_per_TBLS_DATA_STATIC01.dbf

2 Add another data file for NMS_PER_TBLS_DATA tablespace

Check the available disk space on the database server. The maximum size of the
new data file should be smaller than the 25% of the available disk space, and
should be smaller than 32G.
SQL> ALTER TABLESPACE NMS_PER_TBLS_DATA ADD DATAFILE
‘/opt/nmsdb/oradata/data/nms_per_TBLS_DATA_STATIC06.dbf ' size
128M AUTOEXTEND ON NEXT 128M MAXSIZE 32G;
3 Query the data files of NMS_PER_TBLS_IDX

SQL> select file_name from dba_data_files where tablespace_name=’
NMS_PER_TBLS_IDX’

FILE_NAME
------------------------------------------------------------------
--------------
/opt/nmsdb/oradata/idx/nms_per_TBLS_IDX_STATIC.dbf
4 Add another data file for NMS_PER_TBLS_IDX tablespace

Check the available disk space on the database server. The maximum size of the
new data file should be smaller than the 25% of the available disk space, and
should be smaller than 32G.
SQL> ALTER TABLESPACE NMS_PER_TBLS_IDX ADD DATAFILE
‘/opt/nmsdb/oradata/idx/nms_per_TBLS_IDX_STATIC01.dbf’ size 128M
AUTOEXTEND ON NEXT 128M MAXSIZE 32G;
New Redo Log cannot be added

Possible Scenario
Oracle alert log (nms_alert.log under subdirectory /admin/nms/bdump of Oracle
installation directory ) including the following error information.
Thread1 cannot allocate new log.
Checkpoint not complete.
Possible Cause
The size of Redo log is not big enough; the switch between redo logs is too frequent.
Impaction on System
Once this error happens, all the DML/DDL operations to database may hang
temporarily.
Solution
Checking db_writer_processes Value
1 Check db_writer_processes
$sqlplus sys as sysdba
SQL>show parameter;
2 Check the value of db_writer_processes, this value should be smaller than the
number of CPU. For example, if there are four CPUs of this server, set the value
to three.
SQL>alter system set db_writer_processes = 3 scope=spfile;
SQL>create pfile from spfile;
3 Restart the Oracle instance.
Querying Redo Log Status

1 Query the current redo log members
SQL> select member from v$logfile;

MEMBER
------------------------------------------------------------------
/opt/nmsdb/oraESL/oradata/nms/redo03.log
2 Query the status of each redo log

SQL> select group#,members,bytes/1024/1024,status from v$log;

GROUP# MEMBERS BYTES STATUS
1 1 200 INACTIVE
2 1 200 INACTIVE
3 1 200 CURRENT
The result indicates that there are three redo log groups, each group has one
member, and member size is 200M. We need to add two more members and
change the three old member’s size to 500M.
There should be only one redo group in CURRENT status and at least one group
in ACTIVE status.
3 Add more redo log groups and resize old redo log size.
Adding More Redo Log Groups
Uses the following command to adding more redo log groups, and resize the size of
old redo logs.
4 Create two new redo log group
SQL> alter database add logfile group 4
('/opt/nmsdb/oraESL/oradata/nms/redo04.log ') size 500M;


('/opt/nmsdb/oraESL/oradata/nms/redo054.log ') size 500M;
5 Switch redo log to the new redo log group 4

SQL>alter system switch logfile;
SQL>select group#,members,bytes/1024/1024,status from v$log;
6 Wait until the status of group 4 changes into CURRENT, and then switch redo
log to the new redo log group 5.
SQL>alter system switch logfile;
SQL>select group#,members,bytes/1024/1024,status from v$log;

GROUP# MEMBERS BYTES/1024/1024 STATUS
---------- ---------- --------------- ----------------
1 1 200 ACTIVE
2 1 200 ACTIVE
3 1 200 ACTIVE
4 1 500 CURRENT
5 1 500 ACTIVE
7 Wait until the STATUS of the redo log group1, group 2, group 3 is changed into
INACTIVE, then delete the old three redo log group
SQL>alter database drop logfile group 1;
8 Delete the redo log file under /opt/nmsdb/oraESL/oradata/nms/REDO01.LOG,

REDO02.LOG and REDO03.LOG
9 Rebuild redo log group 1, 2 and 3, and change the size of each group to 500M
('/opt/nmsdb/oraESL/oradata/nms/REDO01.LOG') size 500M;
MN9200 Response Unacceptable slow

Possible Scenario
When the user perform any operation in MN9200 server, MN9200 server response is
unacceptable slow.
Probable Cause
The performance of the MN9200 is not good.

Solution
Perform the following operation to collect the performance related logs, and send the
logs to the field support engineer of NEC for further analyze.
Querying Server’s Performance
The following information should be collected:
 Collect the IO information 100 reports at five seconds interval, and output the
result to the specified log
iostat -xcn 5 100 >>/opt/netmantools/log/iostatfile &
 Collect the virtual memory information every five seconds, and output the result
to the specified log
vmstat 5 100 >>/opt/netmantools/log/vmstafile &
 Collect the global average activities for each available processor 100 reports at
five seconds interval, and output the result to the specified log
mpstat 5 100 >>/opt/netmantools/log/mpstatfile &
 Collect the system activity information 100 reports at five seconds interval, and
output the result to the specified log.
sar 5 100 >>/opt/netmantools/log/sarfile &
 Collect active processes on the system and reports statistics based on the
selected output mode and sort order 100 reports at five seconds interval, and
output the result to the specified log.
prstat –a 5 100 >>/opt/netmantools/log/prstatfile &
 Collect the process information

a View the process ID of java, and record all the process ID
ps -ef|grep java
b Check which process uses 2008 port

pfiles PID |grep -i "port: 2008"
For example,
#pfiles 2902|grep -i "port: 2008"
sockname: AF_INET 0.0.0.0 port: 2008

c Collect the data every 5000 ms (that is five seconds), and execute the
command 1000 times
Replace the Procees_ID with the process which uses 2008 port.

/usr/jdk/instances/jdk1.6.0/bin/jstat -gc Process_ID 5000 1000

>>/opt/netmantools/log/$gcfile &
In this example, execute

/usr/jdk/instances/jdk1.6.0/bin/jstat -gc 2902_ID 5000 1000
>>/opt/netmantools/log/$gcfile &
Querying Database Performance on Database Server

 Record the SGA hit ratio
SQL>select a.value + b.value "logical_reads", c.value "phys_reads",
round(100 * ((a.value+b.value)-c.value) / (a.value+b.value))
"BUFFER HIT RATIO" from v$sysstat a, v$sysstat b, v$sysstat c
where a.statistic# = 38 and b.statistic# = 39
and c.statistic# = 40
 Record DB buffer hit ratio

SQL>SELECT a.VALUE + b.VALUE logical_reads, c.VALUE phys_reads,
round((1-c.value/(a.value+b.value)),4) hit_ratio FROM v$sysstat
a,v$sysstat b,v$sysstat c WHERE a.NAME='db block gets' AND
b.NAME='consistent gets' AND c.NAME='physical reads';
 Record library cache hit ratio

SQL>select sum(pins) "Total Pins", sum(reloads) "Total Reloads",
sum(reloads)/sum(pins) *100 libcache from v$librarycache
 Record the dictionary cache hit ratio

SQL>SELECT (1-(SUM(getmisses)/(SUM(gets)+SUM(getmisses))))*100
FROM v$rowcache;
SQL>select parameter, gets,Getmisses ,
getmisses/(gets+getmisses)*100 "miss ratio",
(1-(sum(getmisses)/ (sum(gets)+sum(getmisses))))*100 "Hit ratio"
from v$rowcache where gets+getmisses <>0
group by parameter, gets, getmisses;
 Record data file’s r/w status

SQL>select substr(a.file#,1,2) "#", name, a.status, a.bytes,
b.phyrds, b.phywrts from v$datafile a, v$filestat b where a.file#
= b.file#
 Record redo log buffer hit ratio

SQL>SELECT name, gets, misses, immediate_gets, immediate_misses,
Decode(gets,0,0,misses/gets*100) ratio1,
Decode(immediate_gets+immediate_misses,0,0,
immediate_misses/(immediate_gets+immediate_misses)*100) ratio2
FROM v$latch WHERE name IN ('redo allocation', 'redo copy');
 Collecting Redo log wait Logs

SQL>SELECT name,value FROM v$sysstat
WHERE name=’redo log space requests’;

SQL>Select * from v$session_wait where event=’log buffer wait’;
SQL>Select event,total_waits,time_waited,average_wait from
v$system_event where event like 'log file switch completion%';
 Collect undo segment

SQL>SELECT class,count FROM v$waitstat WHERE class IN (’system
undo header’,'system undo block’, ’undo header’,’undo block’);
SQL>SELECT sum(value) FROM v$sysstat WHERE name IN (‘db block
gets’,’consistent gets’);
 Collect the current locked object

SQL>select
db.object_name,sess.sid,sess.serial#,sess.module,sess.action,
'alter system kill session '''|| sess.sid
||','||sess.serial#||'''' from
v$locked_object obj, dba_objects db, v$session sess
where obj.object_id=db.object_id and obj.session_id=sess.sid
 Collect the current wait object

SQL>select row_wait_obj# from v$SESSION where EVENT in ( 'bufffer
busy waits','db file sequential read','db file scattered
read','enqueue','free buffer waits', 'latch free','log file
parallel write','log file sync');
SQL>select owner, object_name, subobject_name, object_type from
DBA_OBJECTS where data_object_id = &row_wait_obj#
 Collect the current executing SQLat the interval of 10 seconds, and execute six
times
SQL>SELECT osuser, username, sql_text from v$session a, v$sqltext
b where a.sql_address =b.address order by address, piece;
SQL>select s.USERNAME, sq.SQL_TEXT, s.LAST_CALL_ET
from v$open_cursor oc, v$sql sq, v$session s
where oc.HASH_VALUE = sq.HASH_VALUE
and s.SID = oc.SID and s.STATUS = 'ACTIVE';
Collecting Oracle Logs

Collect the following log files:
 Copy all the files under MN9200 running logs directory
$MN9200_installation_folder/logs/* on the application server
 Collect the Oracle alert log nms_alert.log stored at
/opt/nmsdb/admin/nms/bdump/ on the database server
 Collect the Oracle trace log files *.trc stored at /opt/nmsdb/admin/nms/bdump
and /opt/nmsdb/admin/nms/udump on the database server
Collect Veritas related log and configuration file if using Veritas HA
 Copy the Veritas log engine_A.log in the directory /var/VRTSvcs/log
 Copy the Veritas configuration file main.cf in the directory

/etc/VRTSvcs/conf/config
Collect the logs in NMS server
 Export the operation log to CSV or PDF file
 Export the system log to CSV or PDF file
 Export the event to CSV or PDF file
MN9200 Server Crashes

Possible Scenario
The MN9200 server is crashed.
Solution
Collect the following files and send the logs to the field support engineer of
1(C for further analyze.
 The file core in the MN9200 application software installation directory
 The file hs_err_pid*.log in the MN9200 application software installation directory
 All the files under the subdirectory /logs the MN9200 application software
installation directory

Backup Tool Usage
A A
This chapter describes how to use backup tool to back up and restore MN9200
database provided by NEC.
This tool is applicable to RDMBS database.
Preparations
1 MN9200 server has already installed.
2 Copy the latest version of the Backup tool software to the MN9200 server.
Backing up Database
Set environment parameters before backing up database.
Setting Environment Parameters

2 Enter the directory where stores the backup tool.
3 Modify the configuration file set_env.sh according to the actual configuration.
The example file is listed below.
$vi set_env.sh
#!/bin/sh
echo off
#XXX_USER is user account of the XXX database, which is created
during MN900 database installation;
#XXX_PASS is the password of XXX_USER,
#XXX_OUT_FOLDER is the directory where stores the backed up XXX
database files.
NML_USER=nms_nml; export NML_USER;
NML_PASS= nms_nml; export NML_PASS;
NML_OUT_FOLDER=nml255; export NML_OUT_FOLDER;
EML_USER=nms_eml; export EML_USER;

EML_PASS=nms_eml; export EML_PASS;
EML_OUT_FOLDER=eml255; export EML_OUT_FOLDER;
NEL_USER=nms_nel; export NEL_USER;

NEL_PASS=nms_nel; export NEL_PASS;
NEL_OUT_FOLDER=nel255; export NEL_OUT_FOLDER;

58 Appendix A Terms and Acronyms
PER_USER=nms_per; export PER_USER;

PER_PASS=nms_per; export PER_PASS;
PER_OUT_FOLDER=per255; export PER_OUT_FOLDER;
ALM_USER=nms_alm; export ALM_USER;

ALM_PASS=nms_alm; export ALM_PASS;
ALM_OUT_FOLDER=alm255; export ALM_OUT_FOLDER;
EVT_USER=nms_evt; export EVT_USER;

EVT_PASS=nms_evt; export EVT_PASS;
EVT_OUT_FOLDER=evt255; export EVT_OUT_FOLDER;
Backing up NML Database Only

Execute the script:
$./backup_nml.sh
Backing up EML Database Only

Execute the script:
$./backup_eml.sh
Backing up History Alarm Database Only

Execute the script:
$./backup_alm.sh
Backing up Event Database Only

Execute the script:
$./backup_evt.sh
Backing up NEL Database Only

Execute the script:
$./backup_nel.sh
Backing up History Performance Database Only

Execute the script:
$./backup_per.sh
Backing up all the Database

Execute the scrip to backup all the database at one time:
$./backup_all.sh

Appendix A Terms and Acronyms 59
Restoring Database
Stop the MN9200 server before restoring Database.
Stopping MN9200 Server

1 Log on to the database server as root.
2 Go to the <MN9200 application installation folder>/bin directory.
If the MN9200 installed at /opt/nms/SpectravlWave_MN9200_2.5, the
command is:
#cd /opt/nms/SpectravlWave_MN9200_2.5/bin
3 Execute the following commands:

# ./server_stop.sh
Restoring MN9200 Database

Setting Environment Parameters
2 Enter the directory where stores the backup tool.
3 Modify the configuration file set_env.sh according to the actual configuration.
Restoring NML Database Only

Execute the script:
$./restore_nml.sh
Restoring EML Database Only

Execute the script:
$./restore_eml.sh
Restoring History Alarm Database Only

Execute the script:
$./restore_alm.sh
Restoring Event Database Only

Execute the script:
$./restore_evt.sh
When the database is restored, the following message displayed:

Enable constraint...
Complelte restore the database, please check ./evt2540/restore.log
Restoring NEL Database Only

Execute the script:

60 Appendix A Terms and Acronyms
$./restore_nel.sh
Restoring History Performance Database Only

Execute the script:
$./restore_per.sh
Restoring all the Database

Execute the scrip to backup the entire database at one time:
$./restore_all.sh

Spectralwave Mn9200: Maintenance Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Spectralwave Mn9200: Maintenance Guide

Uploaded by

Copyright:

Available Formats

SpectralWave MN9200

System Release 2.5

Doc. Code L2 FL26 2500 10 12 00

NEC Corporation 7-1, Shiba 5-Chome, Minato-Ku, Tokyo 108-8001, Japan

1 System Requirements .......................................................................................3

2 Routine Maintenance .........................................................................................5

3 Database Backup and Restoration.................................................................19

Restoring RDMBS Database.......................................................................................19

4 MN9200 Upgrading .......................................................................................... 31

Maintenance Guide SpectralWave MN9200

A Backup Tool Usage..........................................................................................57