NGCSS SM09 E1 0 NGCSS Routine Maintenance

NGCSS
Routine Maintenance
Training Material
ZTE UNIVERSITY
ZTE University, Dameisha
YanTian District, Shenzhen,
P. R. China
518083
Tel: (86) 755 26778800
Fax: (86) 755 26778999
URL: http://ensupport.zte.com.cn
E-mail: support@zte.com.cn
LEGAL INFORMATION
Copyright © 2006 ZTE CORPORATION.
The contents of this document are protected by copyright laws and international treaties. Any reproduction or distribution of
this document or any portion of this document, in any form by any means, without the prior written consent of ZTE CORPO-
RATION is prohibited. Additionally, the contents of this document are protected by contractual confidentiality obligations.
All company, brand and product names are trade or service marks, or registered trade or service marks, of ZTE CORPORATION
or of their respective owners.
This document is provided “as is”, and all express, implied, or statutory warranties, representations or conditions are dis-
claimed, including without limitation any implied warranty of merchantability, fitness for a particular purpose, title or non-in-
fringement. ZTE CORPORATION and its licensors shall not be liable for damages resulting from the use of or reliance on the
information contained herein.
ZTE CORPORATION or its licensors may have current or pending intellectual property rights or applications covering the subject
matter of this document. Except as expressly provided in any written license between ZTE CORPORATION and its licensee,
the user of this document shall not acquire any license to the subject matter herein.
ZTE CORPORATION reserves the right to upgrade or make technical change to this product without further notice.
Users may visit ZTE technical support website http://ensupport.zte.com.cn to inquire related information.
The ultimate right to interpret this product resides in ZTE CORPORATION.
Publishing Date (MONTH/DATE/YEAR) : 11/10/2010

Contents
Routine Maintenance.................................................... 1
1Overview of Routine Maintenance .............................. 3
1.1Introduction to Routine Maintenance ......................................... 3
1.2Requirements for the Maintenance Personnel.............................. 4
1.3Precautions in Routine Maintenance .......................................... 5
1.4Common Maintenance Methods ................................................ 6
1.5Maintenance Tools .................................................................. 7
2Daily Maintenance ...................................................... 9
2.1Universal Daily Maintenance Items............................................ 9
2.1.1Checking the Equipment Room......................................... 9
2.1.2Checking Hardware Indicators.........................................10
2.1.3Checking Disks .............................................................11
2.1.4Checking Database Space...............................................12
2.2OMMP Checks .......................................................................13
2.2.1OMMP Maintenance Task List...........................................13
2.2.2Checking SMAP Alarms ..................................................13
2.2.3Checking Daily Peak Traffic .............................................13
2.3SIP Proxy Server Check..........................................................14
2.4Oracle Database Server Check ................................................14
2.5Parlay Server Check...............................................................14
2.6EBASE Server Check ..............................................................15
2.7Notify Scheduler Check ..........................................................15
2.8UMA Check ...........................................................................15
2.9Mail Processor Check .............................................................15
2.10Mail Clear Server Check ........................................................15
2.11Smart Call Alert Check .........................................................15
2.12VGW Check.........................................................................15
2.13Media Server (CMS9000/3000) Checks ...................................16
2.13.1Media Server Maintenance Task List ...............................16
2.13.2Checking Media Server Hardware Indicators ....................16
2.13.3Checking Media Server Alarms ......................................16
2.14Network Storage Check ........................................................17
2.15Web Server Check ...............................................................17
2.16Statistics Server Checks .......................................................17
2.17Sybase Database Server Check..............................................18
2.18Service Flow Checks.............................................................18
2.18.1Testing Main Service Flows............................................18
2.18.2Checking Basic Service Functions...................................18
2.19Firewall Check .....................................................................18
2.20Layer 3 Switch and GRE Router Check ....................................19
3Weekly Maintenance ................................................ 21
3.1Universal Weekly Maintenance Items........................................21
3.1.1Checking NICs ..............................................................21
3.1.2Checking System Time...................................................22
3.1.3Checking Core Files .......................................................24
3.1.4Checking Disk Space......................................................24
3.1.5Checking the CPU Usage ................................................24
3.1.6Checking the Memory Usage ...........................................25
3.1.7Checking the Swap Space Usage .....................................25
3.1.8Checking Disk I/O .........................................................25
3.1.9Checking Packet Loss.....................................................26
3.1.10Checking the Database Running Status...........................27
3.1.11Checking the Cluster ....................................................28
3.1.12Checking WINNTTCP Communication..............................29
3.1.13Checking the Connection Between the Database Server
and the Database Client ...................................................30
3.1.14Checking IMP Processes ...............................................34
3.1.15Checking the NFS Client ...............................................35
3.2OMMP Checks .......................................................................36
3.2.2Checking WINNTTCP Communication ...............................37
3.2.3Checking the Connection Between the OMMP and the
Database........................................................................37
3.2.4Checking Scheduled Tasks ..............................................38
3.2.5Checking the OMMP Process ...........................................39
3.3SIP Proxy Server Checks ........................................................40
3.3.1SIP Proxy Maintenance Task List......................................40
3.3.2Checking SIP Proxy Processes .........................................40
3.4Oracle Database Server Checks ...............................................41
3.4.1Oracle Database Server Maintenance Task List ..................41
3.4.2Checking Automatic Database Backup ..............................42
3.4.3Checking the Resource Usage of Database Processes .........43
3.4.4Checking Database Jobs .................................................43
3.5Parlay Server Checks .............................................................43
3.5.1Parlay Server Maintenance Task List.................................43
3.5.3Checking the Connection Between the Parlay Server and
the Database ..................................................................45
3.5.4Checking Scheduled Tasks ..............................................45
3.5.5Checking the Parlay Server Process .................................46
3.6EBASE Server Checks.............................................................47
3.6.1EBASE Server Maintenance Task List ................................47
3.6.2Checking the EBASE Processes........................................47
3.7Smart Call Alert Checks..........................................................48
3.7.1Smart Call Alert Maintenance Task List .............................48
3.7.3Checking Smart Call Alert Service Processes .....................49
3.8VGW Checks .........................................................................49
3.8.1VGW Maintenance Task List ............................................49
3.8.3Checking the VGW Process .............................................50
3.9Media Server (CMS9000/3000) Checks .....................................51
3.9.1Media Server Maintenance Task List .................................51
3.9.2Checking the CPU Usage of Media Server..........................51
3.9.3Checking the Bandwidth Usage of the MPC Board ..............53
3.10Network Storage Checks .......................................................54
3.10.1Network Storage Maintenance Task List ..........................54
3.10.2Checking Inodes..........................................................54
3.10.3Checking Disk Space of Network Storage ........................54
3.11Web Server Checks ..............................................................55
3.11.1Web Server Maintenance Task List .................................55
3.11.2Checking Web Login.....................................................55
3.11.3Checking Tomcat Processes...........................................56
3.11.4Checking JBoss Processes .............................................57
3.12Notify Scheduler Check.........................................................57
3.13UMA Checks ........................................................................58
3.13.1UMA Maintenance Task List ...........................................58
3.13.2Checking UMA Processes ..............................................59
3.14Mail Processor Check............................................................61
3.14.1Mail Processor Maintenance Task List..............................61
3.14.2Checking WINNTTCP Communication..............................61
3.14.3Checking Mail Processor Processes .................................62
3.14.4Checking the Connection Between the Mail Processor
and the Network Storage ..................................................62
3.15Mail Clear Server Check ........................................................63
3.17Checking Power Supplies of Firewall .......................................65
4Monthly Maintenance ............................................... 67
4.1Universal Monthly Maintenance Items.......................................67
4.1.1Checking Power Supplies ................................................67
4.1.2Checking Operating System Patches ................................68
4.1.3Freeing Up Disk Space ...................................................68
4.1.4Checking Operating System Logs.....................................70
4.1.5Checking Cluster Logs....................................................71
4.1.6Checking Database Logs.................................................71
4.1.7Checking IMP Logs ........................................................74
4.2OMMP Checks .......................................................................75
4.2.2Checking OMMP Logs .....................................................75
4.2.3Backing Up Application Programs and Log Files..................76
4.3Parlay Server Checks .............................................................76
4.3.1Parlay Server Maintenance Task List.................................76
4.3.2Checking Parlay Server Logs ...........................................77
4.4SIP Proxy Server Checks ........................................................78
4.4.1SIP Proxy Server Maintenance Task List............................78
4.4.2Checking SIP Proxy Logs ................................................78
4.5EBASE Server Checks.............................................................79
4.5.1EBASE Server Maintenance Task List ................................79
4.5.2Checking EBASE Logs ....................................................79
4.6Smart Call Alert Checks..........................................................80
4.6.1Smart Call Alert Maintenance Task List .............................80
4.6.2Checking Smart Call Alert Logs .......................................80
4.6.3Backing Up Application Programs and Configuration
Files ..............................................................................81
4.7VGW Checks .........................................................................81
4.7.1VGW Maintenance Task List ............................................81
4.7.2Checking VGW Logs.......................................................82
4.7.3Backing Up Application Programs and Configuration
Files ..............................................................................82
4.7.4Checking the Monitoring Program ....................................82
4.8Oracle Database Server Checks ...............................................83
4.9Web Server Checks................................................................83
4.9.1Web Server Maintenance Task List ...................................83
4.9.2Checking Web Service Logs ............................................83
4.10UMA Checks ........................................................................86
4.10.1UMA Maintenance Task List ...........................................86
4.10.2Checking UMA Logs .....................................................86
4.10.3Checking the Monitoring Program ..................................87
4.11Mail Processor Check............................................................88
4.11.1Mail Processor Maintenance Task List..............................88
4.11.2Checking Mail Processor Logs ........................................88
4.11.3Backing Up Application Programs and Log Files ................88
4.11.4Checking Connections Between the Mail Processor and
Other Servers .................................................................89
4.12Mail Clear Server Checks ......................................................89
4.13Sybase Database Server Checks ............................................90
4.15Firewall Checks ...................................................................90
4.15.1Firewall Maintenance Task List .......................................90
4.15.2Checking Firewall Logs .................................................91
4.15.3Checking Firewall Alarms ..............................................92
4.16Layer 3 Switch and GRE Router Checks...................................93
4.16.1Layer 3 Switch and GRE Router Maintenance Task
List ................................................................................93
4.16.2Checking Interfaces .....................................................93
4.16.3Checking VLANs ..........................................................94
4.16.4Checking the Routing Table ...........................................95
4.16.5Checking Big-IP Logs of the F5 Device ............................96
5Quarterly Maintenance ............................................. 99
5.1Checking Power Connections ...................................................99
5.2Cleaning Cabinets..................................................................99
5.3Removing Foreign Objects .................................................... 100
5.4Checking Ground Wires ........................................................ 100
5.5Testing Grounding Resistance ................................................ 100
5.6Checking Fans..................................................................... 101
5.7Cleaning Tape Drives............................................................ 101
Routine Maintenance
After you have completed this course, you
will be able to:

>> Understand Common Maintenance
Methods of the ZTE NGCSS
>> Understand Routine Maintenance of
the ZTE NGCSS
>> Understand Exception Handling of the
ZTE NGCSS
Confidential and Proprietary Information of ZTE CORPORATION 1

NGCSS Routine Maintenance
This page is intentionally blank.
2 Confidential and Proprietary Information of ZTE CORPORATION

�1 Overview of Routine
Maintenance
After you completing this chapter, you will know:
>> Introduction to Routine Maintenance
>> Requirements for the Maintenance Personnel
>> Precautions in Routine Maintenance
>> Common Maintenance Methods
>> Maintenance Tools
1.1 Introduction to Routine

Maintenance
Definition Routine maintenance refers to periodic checks on the equipment.
Maintenance personnel correct discovered problems in time for the
purpose of discovering potential problems, preventing accidents
and eliminating faults as soon as possible.
Classification Routine maintenance is classified into four types:
¡ Daily maintenance
¡ Weekly maintenance
¡ Monthly maintenance
¡ Quarterly maintenance
The purpose of daily maintenance is to find existing problems or
potential faults.
Daily maintenance involves the following scopes:
¡ Equipment room environment (temperature and humidity)
¡ Software alarms from independent OAMs or the unified net-
work management system
¡ Hardware indicators
¡ Key service flows
¡ Basic service functions
¡ Disk usage
¡ Core files
¡ Scheduled tasks
All daily maintenance tasks are completed within one hour.
Weekly maintenance focuses on performance measurement and
analysis and involves the following scopes:

¡ CPU usage
¡ Memory usage
¡ Swap usage
¡ Disk I/O
¡ Database space usage
¡ Network traffic
¡ Cluster
¡ Monitoring process
¡ System backup
¡ Service volume in busy hour
Monthly maintenance involves network check, log check and en-
vironment check.
Quarterly maintenance involves checking fans, testing grounding
resistance, cleaning tape drives, and cleaning the equipment
room.
In terms of component types, the routine maintenance of NGCSS
is divided into hardware maintenance and software maintenance.
Hardware components include database servers, application
servers, voice playing device, and statistics server. Software
includes operating systems, databases and service systems.
Application servers include:
¡ OMMP
¡ SIP Proxy Server
¡ Parlay Server
¡ Notify Scheduler
¡ UMA
¡ Mail Processor
¡ Mail Clear Server
¡ Media Server
¡ Web server
¡ Smart Call Alert
¡ VGW
1.2 Requirements for the

Maintenance Personnel
Background The maintenance personnel must be familiar with:
Information
¡ Ethernet and TCP/IP
¡ Oracle, Sybase, and SQL Server databases
¡ Basic operations on servers

Chapter 1 Overview of Routine Maintenance
Product ¡ Hardware structure and performance parameters of the NGCSS

Knowledge system
¡ NGCSS position in the network
¡ Message interactions between NGCSS system devices and
other devices
¡ Service flows of the NGCSS system
Operation To ensure efficiency and prevent improper operations, the main-
Requirements tenance personnel must:
¡ Be trained and qualified
¡ Be familiar with operation flows of the NGCSS system
¡ Comply with operation regulations and industry safety regula-
tions to ensure personal and device safety
1.3 Precautions in Routine

Maintenance
Keep the optimum temperature (15℃~25℃) and humidity
(30%~70%) inside the equipment room. Make sure the equip-
ment room is clean, dust proof and damp proof. Make sure there
are no rats or insects in the equipment room.
Caution:
High temperature may cause system server failure and results in
system failure.
Make sure that the primary power supply is stable and reliable.
Check the grounding and lightning protection ground regularly.
Check the lightning protection system before the thunderstorm
season and after each thunderstorm to make sure facilities are in
good condition.
Equipment room illumination must meet the maintenance require-
ment. In normal days, damaged lights should be repaired in time
to prevent illumination from being inconvenient to the mainte-
nance.
Establish a sophisticated equipment room maintenance system
and regulate maintenance personnel's daily work. Make detailed
on-duty logs to record details of daily operation, version condi-
tion, data change, upgrade and problem handling for the system
for analysis and handling when problems occur later. There must
be a duty-shifting record for responsibility identification.
Routine maintenance requires common tools and meters such as
screwdrivers (straight screwdriver and cross screwdriver), signal-
ing analyzer, cable clamp, multimeter, telephone wire and network
cable. Check the meters periodically to ensure their accuracy.
Often check the spare parts to ensure the inventories are avail-
able and in good condition, and protect them against moisture
and mildew. Spare parts must be stored separately from bad parts

and marked with a sign. Commonly used parts must be supplied

in time before being used up.
Software and documents that may be used during maintenance
should be placed in the specified positions and can be accessed
when they are required.
Maintenance personnel should be trained before beginning to
work. They must have some knowledge of the equipment and
related networks. Maintenance personnel must follow the instruc-
tions in the relevant NGCSS device manual, and wear anti-static
wrist straps when touching device hardware to avoid accident
due to human factors. Maintenance personnel should have a
strict attitude towards work and proficient maintenance skills and
improve themselves by continuous study.
Maintenance personnel are not allowed to play games, visit the In-
ternet, or install, run or copy unrelated software on the computers
in the equipment room, or use the computers for other purposes.
1.4 Common Maintenance

Methods
¡ Contrast
Contrast is a simple method. Compare faulty data configura-
tions or devices with normal configurations or devices to find
the differences between them and to locate the fault cause.
¡ Replacement
Replacement is a simple yet good way for troubleshooting. For
a hardware fault, replace the suspect component with a new
one of the same functions or the same type, which is the best.
If the fault is eliminated after replacement, this indicates the
replaced component is faulty.
¡ Minimum System Configuration
Minimum system configuration is to remove other hardware
devices to reserve the necessary parts of the system only and
check if the fault persists in the minimum system. If the fault
persists, this indicates the existing hardware is faulty. If the
fault disappears, add other devices one by one to check which
device causes the fault, and replace the faulty device.
¡ Signaling Analysis
Signaling analysis is the most basic method used in inter-office
and intra-office service interconnection and debugging. Due
to the uniform standard of signaling, you can locate a fault
accurately through signaling analysis.
¡ Failure Code Analysis
Failure code analysis is used to diagnose local service faults.
Every failure in the service has a failure code and cause value.
The interpretation of a failure code helps to locate the fault
cause.

Chapter 1 Overview of Routine Maintenance
1.5 Maintenance Tools

Tools used for routine maintenance of NGCSS include:
¡ SUSE Linux operating system commands, Sybase database
commands, Oracle database commands and SQL Server data-
base commands
¡ The protocol analyzer Ethereal
¡ The ZTE signaling tracing tool MTrace
¡ The service management system (SMS)


�2 Daily Maintenance
>> Universal Daily Maintenance Items
>> OMMP Checks
>> SIP Proxy Server Check
>> Oracle Database Server Check
>> Parlay Server Check
>> EBASE Server Check
>> Notify Scheduler Check
>> UMA Check
>> Mail Processor Check
>> Mail Clear Server Check
>> Smart Call Alert Check
>> VGW Check
>> Media Server (CMS9000/3000) Checks
>> Network Storage Check
>> Web Server Check
>> Statistics Server Checks
>> Sybase Database Server Check
>> Service Flow Checks
>> Firewall Check
>> Layer 3 Switch and GRE Router Check
2.1 Universal Daily Maintenance

Items
2.1.1 Checking the Equipment Room
Purpose Check the equipment room to make sure it meets requirements.

Instruction The maintenance personnel should check the referenced standards
according to the following guidelines:
¡ Use the hygrothermograph to check if the temperature and
humidity in the equipment room meet requirements.
¡ Check if the equipment room floor is clean.
¡ Check if air ventilation is normal.

¡ Check if devices in the equipment room are properly grounded.

¡ Check if cabinet surface is clean and if there is any foreign
object.
Standard for The equipment room should meet the following requirements:
Reference
¡ Temperature : 15℃~30℃ (20±2℃ is the best), with a fluctua-
tion of less than 3℃ per hour.
¡ Relative humidity: 30%~70%, with a fluctuation of less than
3% per hour.
¡ Clean floor with no carpet. The density of dust larger than 5μm
is less than 3×104/m3.
¡ Enough space with proper air ventilation.
¡ Devices are properly grounded and without electrostatic dis-
charge.
¡ Clean rack surface, without any foreign matter.
¡ No user guide or report paper in the main equipment room.
Exception ¡ If the temperature or humidity exceeds the range required,
Handling adjust the air conditioner in the equipment room.
¡ If there is any dirt, dust or rust, clean it.
¡ Insufficient space in the equipment room may adversely affect
indoor ventilation. In this case, adjust cabinet position accord-
ing to placement requirements.
¡ If a device is not grounded, ground it as required to avoid elec-
trostatic discharge.
¡ Remove foreign object from cabinet surface if there is any.
2.1.2 Checking Hardware Indicators
Purpose Make sure that the hardward components are working normally.
Instruction 1. Check the system power indicator of the system.
2. Check the health indicator of the system.
3. If there is an LED panel, check and record the system status
code displayed on the LED panel.
Standard for ¡ The power indicator, health indicator and LED panel is all green
Reference or blue.
¡ Orange or yellow indicates minor faults.
¡ Red indicates major or critical faults.
¡ If the power indicator is unlit, this indicates that the system
has a power failure.
¡ If the power indicator is yellow and all the other indicators are
unlit, this indicates that the system had a power failure and
devices are not restarted.
Refer to related manuals to know the position and meaning of each
indicator.

Chapter 2 Daily Maintenance
Exception ¡ If the system has a power failure, check the AC and DC power
Handling supplies, UPSs, and inverters. Contact power system engi-
neers.
¡ If the indicators or the LED panel are orange or yellow, perform
the following steps:
i. Check the LED panel. If error code is displayed on the LED
panel, record error code and contact ZTE technical support
engineers.
ii. Check power supplies. If either of the two power supplies
fails, perform the following steps:
– If the power connector is loose, re-plug the power con-
nector.
– If the switch of the rack is off, contact power system
engineers and turn on the switch when the power sys-
tem is normal.
– If the power system is faulty, contact power system
engineers.
iii. If the disk indicator is orange or red, this indicates that
the hard disk is faulty or damaged. Contact ZTE technical
support engineers.
iv. If the fans for the power supplies or in the chassis do not
work normally, contact ZTE technical support engineers.
v. If the temperature or humidity in the equipment room is
abnormal, contact air conditioner engineers.
vi. For other faults, contact ZTE technical support engineers.
¡ If the indicator is red, this indicates that major or critical faults
occur. Contact ZTE technical support engineers.
2.1.3 Checking Disks
Purpose Check whether the local disk and all disks in the disk array are
normal.
Instruction 1. Observe whether the color and flashing of the indicator of the
local disk are normal.
2. Observe whether the color and flashing of the disk indicators
in the disk array are normal.
3. Log in to each server as user "root" and run the following com-
mand:
� SUSE Linux
Run the #fdisk -l command.
Check disk partitions.
� Windows
To check the disk in a single-server configuration, perform
the following steps:
a) Select Start > Settings > Control Panel > Ad-
ministrative Tools > Computer Management. In

the Computer Management Window, select Storage

> Disk Management. Check the status of every
partition.
b) Select Start > Settings > Control Panel > Admin-
istrative Tools > Services and make sure that the
status of the Logical Disk Manager is "Started".
In a two-server cluster configuration, the disks on only one
server are in "Healthy" status.
Standard for ¡ The color and flashing of the indicator of the local disk are
Reference normal.
¡ The color and flashing of the disk indicators in the disk array
are normal.
¡ In a single-server configuration, the disks are in "Healthy" sta-
tus.
In a two-server configuration, the disks on only one server are
in "Healthy" status.
Exception Contact ZTE technical support engineers.
Handling
2.1.4 Checking Database Space
Purpose Make sure that the Oracle, Sybase and SQL Server database has
sufficient space.
Instruction ¡ Oracle
i. Launch the Oracle client program, and then select Enter-
prise Manager Console.
ii. Select a database service name, for example, the database
service for the OMMP is ZX192_200_26_133.
iii. Click + and log in using your username and password.
iv. Click Tablespace in Storage to view the usage of every
tablespace.
¡ Sybase
Log in to the server as user "root" and run the following com-
mand:
#cd /opt/sybase/OCS-12_0/bin/
#isql -S<database name> -Usa -P<password>
1>sp_helpdb
2>go
View the database name list and then run the following com-
mand:
1>sp_helpdb <database name>
2>go
¡ SQL Server
After you connect to the SQL Server database, enter the fol-
lowing:

create table tmp(name varchar(50),rows int,reserved varchar(50),data

varchar(50),index_size varchar(50),unused varchar(50))insert into tmp
(name,rows,reserved,data,index_size,unused) exec sp_msforeachTable @C
ommand1="sp_spaceused '?'"--sp_spaceused 't_vehicle'
select * from tmp order by data desc;
Standard for The space usage of the database is below 80%.
Reference
Exception Contact ZTE technical support engineers for database expansion.
Handling
2.2 OMMP Checks

2.2.1 OMMP Maintenance Task List
During daily maintenance on the NGCSS, you need to perform rou-

tine checks listed below on the OMMP.
Daily Maintenance
No. Operation Guide
Item
Hardware alarm
1 2.1.2
indicator
2 SMAP alarm 2.2.2
Maximum daily call

3 2.2.3
traffic
2.2.2 Checking SMAP Alarms
Purpose Discover the potential problems of the intelligent network by

checking SMAP alarms so that you can handle faults in time and
keep the system running normally.
Instruction Click in the main interface of the SMAP and observe and record
alarms.
Standard for No alarms are displayed.
Reference
Exception If there are any alarms of level 3 or above, check the alarms and
Handling handle accordingly.
2.2.3 Checking Daily Peak Traffic
Purpose Make sure that the system capacity can satisfy the daily peak traf-
fic.
Instruction In the SMS system, select System > System Trace. On the
Trace Set dialog box, select Service Call, select VMS service

from the list below and click Trace. In the trace results, use
to move along the curve. Select a date and
an hour range to show the curve for the traffic during this time
period. Select Show value to show the specific value at a point
of the curve.
Standard for The daily peak traffic cannot exceed the 80% of the system ca-
Reference pacity.
Exception If the daily peak traffic often exceeds 80% of the system capacity
Handling on non-holiday days, expand the system capacity. If the daily peak
traffic often exceeds 80% of the system capacity on holiday days,
be prepared for possible failures.
Caution:
Check the daily peak traffic on each Parlay Server.
2.3 SIP Proxy Server Check

During daily maintenance of the SIP proxy server, you need to
check only the power supply to the server. For details, refer to
2.1.2.
2.4 Oracle Database Server

Check
tine checks listed below on the Oracle database server.
Daily Maintenance
No. Operation Guide
Item
Hardware alarm
1 2.1.2
indicator
2 Disk status 2.1.3
3 Database space 2.1.4
2.5 Parlay Server Check

During daily maintenance of the Parlay Server, you need to check
only the power supply to the server. For details, refer to 2.1.2.

2.6 EBASE Server Check

During daily maintenance of the EBASE server, you need to check
2.7 Notify Scheduler Check

During daily maintenance of the Notify Scheduler, you need to
2.1.2.
2.8 UMA Check

During daily maintenance of the UMA, you need to check only the
power supply to the server. For details, refer to 2.1.2.
2.9 Mail Processor Check

During daily maintenance of the Mail Processor, you need to check
2.10 Mail Clear Server Check

During daily maintenance of the Mail Clear server, you need to
2.1.2.
2.11 Smart Call Alert Check

During daily maintenance of the Smart Call Alert, you need to
2.1.2.
2.12 VGW Check

During daily maintenance of the VGW, you need to check only the
power supply to the server. For details, refer to 2.1.2.

2.13 Media Server (CMS9000/3000)

Checks
2.13.1Media Server Maintenance Task
List

tine checks listed below on the media server.
Daily Maintenance
No. Operation Guide
Item
Hardware alarm
1 2.13.2
indicator
2 Media server alarm 2.13.3
2.13.2Checking Media Server

Hardware Indicators
Purpose Make sure that no hardware alarms occur on the CMS9000/3000.

Instruction 1. Check the three alarm indicators on the front panel of the
CMS9000/3000: Critical Alarm, Major Alarm, and Minor Alarm.
2. Check the status indicators of the SCC/MPC board, including
PWR, STAT, and LNK.
Standard for ¡ Three indicators on the front panel of the CMS9000/3000
Reference
� If the Critical Alarm indicator is red, there are critical
alarms.
� If the Major Alarm indicator is red, there are major alarms.
� If the Minor Alarm indicator is yellow, there are minor
alarms.
¡ SCC/MPC board
� If the PWR indicator is red, there are power faults.
� If the STAT indicator or LNK indicator is red, the SCC/MPC
board is faulty.
Exception View alarms through the OAM console of the media server and
Handling contact ZTE technical support engineers.
2.13.3Checking Media Server Alarms
Purpose Make sure that the CMS9000/3000 works normally.

Instruction 1. In the IE address bar, enter "http://xxx.xxx.xxx.xxx".

"xxx.xxx.xxx.xxx" indicates the address of the maintenance
console of the SCC board.
2. Enter admin as the user ID and convedia as the password.
3. Select Fault Mgt > Alarms > Events List.
4. Click Active to display alarms, as shown below.
Standard for If the CMS9000/3000 works normally, no alarms are displayed.

Reference
Handling
2.14 Network Storage Check

During daily maintenance of the Network Storage, you need to
2.1.2.
2.15 Web Server Check

During daily maintenance of the Web server, you need to check
2.16 Statistics Server Checks

During daily maintenance of the statistics server, you need to
2.1.2.

2.17 Sybase Database Server

Check
tine checks listed below on the Sybase database server.
Daily Maintenance
No. Operation Guide
Item
Hardware alarm
1 2.1.2
indicator
2 Disk status 2.1.3
3 Database space 2.1.4
2.18 Service Flow Checks

2.18.1Testing Main Service Flows
Purpose Make sure that the main service flows are normal. The main ser-
vice flows are the host flow and guest flow.
Instruction 1. Leave a message through the guest flow.
2. Retrieve the message through the host flow.
Standard for Service flows are normal.
Reference
Exception Trace SLP/SIP messages through the Mtrace tool and provide ZTE
Handling technical support engineers with them for analysis.
2.18.2Checking Basic Service

Functions
Purpose Make sure that basic service functions are normal.

Instruction Test the basic functions of the NGCSS service.
Standard for Basic service functions are normal.
Reference
Exception Trace SLP/SIP messages through the Mtrace tool and provide ZTE
Handling technical support engineers with them for analysis.
2.19 Firewall Check

During daily maintenance of the firewall, you need to check only
the power supply to the server. For details, refer to 2.1.2.

2.20 Layer 3 Switch and GRE

Router Check
During daily maintenance of the layer 3 switch and GRE router,
you need to check only the power supply to the server. For details,
refer to 2.1.2.


�3 Weekly Maintenance
>> Universal Weekly Maintenance Items
>> OMMP Checks
>> SIP Proxy Server Checks
>> Oracle Database Server Checks
>> Parlay Server Checks
>> EBASE Server Checks
>> Smart Call Alert Checks
>> VGW Checks
>> Media Server (CMS9000/3000) Checks
>> Network Storage Checks
>> Web Server Checks
>> Notify Scheduler Check
>> UMA Checks
>> Mail Clear Server Check
>> Checking Power Supplies of Firewall
3.1 Universal Weekly

Maintenance Items
3.1.1 Checking NICs
Purpose Make sure that all NICs work normally.

Instruction Log in to each server as user "root" and run the following com-
mand:
¡ SUSE Linux
Run the #ifconfig -a command.
The settings of each NIC are displayed. The status of each NIC
is UP or RUNNING.
If a server is used for BOND, check which NIC's indicator is
flashing when data is read.
¡ Windows

i. Select Start > Run. Enter cmd in the text box and click
OK. In the CMD window, enter ipconfig and make sure
that the local address is not changed.
ii. If a server is teamed, check which NIC's indicator is flashing
when data is read.
Standard for ¡ SUSE Linux
Reference
Run the #ifconfig -a command. The settings of each NIC are
displayed. The status of each NIC is UP or RUNNING.
If a server is used for BOND, check which NIC's indicator is
flashing when data is read.
¡ Windows
Enter ipconfig and make sure that the local address is not
changed.
If a server is teamed, check which NIC's indicator is flashing
when data is read.
Exception If there are exceptions, contact ZTE technical support engineers.
Handling
3.1.2 Checking System Time
Purpose Make sure that each server is consistent with the clock server in
system time.
Instruction ¡ SUSE Linux
i. Log in to each server as user "root".
ii. Run the following command:
#date
The system time of the server is displayed.
¡ Windows
Select Start > Run. Enter cmd in the text box and click OK.
� In the CMD window, run the time command. The current
time of the server is displayed, and you are prompted to
enter a new time. If you do not want to change the time,
press Enter directly.
� In the CMD window, run the date command. The current
date of the server is displayed, and you are prompted to
enter a new date. If you do not want to change the date,
press Enter directly.
Standard for Each server is consistent with the clock server in system date and
Reference time.
Exception ¡ Contact ZTE technical support engineers.
Handling
¡ Synchronize System Time
� Synchronize time between a SUSE Linux NTP client and the
NTP clock server.
a) Log in to the SUSE Linux NTP client as user "root".
b) Ping the IP address of the NTP clock server.

Chapter 3 Weekly Maintenance
If the following information is returned, the network

connection is normal.
Reply from <clock server IP address>: bytes=32 time
If not, the network connection is faulty.
c) After the network connection is restored, check whether
the client can connect to the clock server normally.
#ntpq -p
d) Run the following command to synchronize time:
#ntpdate <clock server IP address>
� Synchronize time between a Windows client and the clock
server.
a) Check whether the Windows time service on the Win-
dows client is set to automatic startup.
Select Start > Run. Enter services.msc in the text
box. Check whether the startup type of the Windows
Time service is Automatic.
If not, double-click Windows Time. In the displayed
dialog box, select Automatic from the Startup Type
pull-down list box and click OK.
b) Check whether the network connection between the
Windows client and the clock server is normal.
Select Start > Run. Enter cmd in the text box and
press Enter. The CMD window is displayed. Ping the
IP address of the clock server.
If the following information is returned, the network
connection is normal.
Reply from <clock server IP address>: bytes=32 time
If not, the network connection is faulty.
c) After the network connection is restored, restart the
Windows time service.
In the CMD window, run the following command:
net stop w32time && net start w32time
Select Start > Run. Enter services.msc in the text
box. Right-click the Windows Time service. From
the shortcut menu, click Stop First and then Start to
restart the Windows Time service. The system auto-
matically synchronizes the time with the clock server.
d) When the time difference between the Windows client
and the clock server is larger than 12 hours, the sys-
tem cannot synchronize time automatically. You must
perform the following steps to correct time:
Click the time icon in the lower right corner of the screen
and set the correct date and time.

3.1.3 Checking Core Files
Purpose Make sure that each module runs normally.

Instruction Log in to each server as user "root".
On the Linux operating system, run the following command:
#find / -name core -print
Check whether core files exist.
Standard for No core dump files are displayed.
Reference
Exception Run the #ls -al command to check the generation time of core
Handling dump files. Back up latest core dump files and contact ZTE tech-
nical support engineers.
3.1.4 Checking Disk Space
Purpose Make sure that there is sufficient space on the storage device of
each server.
mand:
¡ SUSE Linux
#df -k
Check every file system that the command lists.
¡ Windows
Check the space usage of every partition in the My Computer
window.
Reference
The space usage of the /usr and /opt file systems is below
90%. The space usage of any other file system is below 80%.
¡ Windows
The free space of the root partition (normally, drive C) is
greater than 2 GB. The space usage of any other partition is
below 80%.
Exception Delete useless files, or contact ZTE technical support engineers for
Handling capacity expansion.
3.1.5 Checking the CPU Usage
Purpose Check whether the CPU usage is within the normal range.
mand:
¡ SUSE Linux
#top

¡ Windows
Press Ctrl+Alt+Del. Click the Performance tab in the Win-
dows Task Manager window. The CPU Usage area shows
the current CPU usage.
Standard for The CPU usage is below 70%.
Reference
Exception If the CPU usage is greater than 70%, observe whether the CPU
Handling usage falls when the traffic is low. If the CPU usage keeps high,
record the average CPU usage, peak value and processes that
maintain a high use of CPU resources. Contact the vendor of the
corresponding server or ZTE technical support engineers.
3.1.6 Checking the Memory Usage
Purpose Make sure that the memory usage is normal.

mand:
¡ SUSE Linux
#cat /proc/cpuinfo
This command shows the memory usage.
¡ Windows
Press Ctrl+Alt+Del. Click the Performance tab in the Win-
dows Task Manager window. The Physical Memory area
shows the current memory usage.
Standard for The size of idle memory is greater than 50 MB or the memory
Reference usage is below 85%.
Exception If the size of idle memory is smaller than 50 MB or the memory us-
Handling age is greater than 85%, contact ZTE technical support engineers.
3.1.7 Checking the Swap Space Usage
Purpose Make sure that the space usage of the swap partition is normal.
mand:
#top
Standard for The swap partition is not used. The system runs normally.
Reference
Exception Save error records and contact ZTE technical support engineers.
Handling
3.1.8 Checking Disk I/O
Purpose Make sure that disk I/O is normal.

Instruction 1. Log in to each server as user "root".

2. Run the following command:

� SUSE Linux
#iostat -d -x 10 100
� Windows
a) Select Start > Settings > Control Panel > Admin-
istrative Tools > Performance.
b) Select Performance Logs and Alerts. Right-click
Counter Logs and select New Log Settings. In the
displayed dialog box, enter the name of the log.
c) Click the General tab. Click Add Counters. In the
Add Counters dialog box, select Physical Disk from
the Performance object pull-down list box.
d) Select All instances.
e) Select Select counters from list. From the list below,
select Current Disk Queue Length and click Add; se-
lect % Disk Time and click Add; select Disk Trans-
fer/sec and click Add.
f) Click Close. Click the Log Files tab. From the Log
file type pull-down list box, select Text File (Comma
delimited). Click Configure. Keep default settings for
other attributes.
g) Keep default settings on the Schedule tab. Click OK.
h) After the counters run for an hour when the system
is busy, select Performance Logs and Alerts >
Counter Logs. Right-click the log and select Stop.
Reference
In the output of the iostat command, the %util field shows
disk I/O. If disk I/O keeps above 85%, this indicates that the
disk is busy.
¡ Windows
The Current Disk Queue Length counter reflects the perfor-
mance of a disk by counting the read and write requests on
the disk. When the value of the % Disk Time counter keeps
greater than 90%, but the values of the Disk Transfer/sec
counter and the Current Disk Queue Length counter are
smaller than 60, this indicates that I/O bottleneck occurs.
When the value of the % Disk Time counter keeps greater
than 90%, the value of the Current Disk Queue Length
counter is 1.5 to 2 times the number of physical disks, and
the value of the Disk Transfer/sec counter is greater than
60, this indicates that the I/O of the disk is busy.
Handling
3.1.9 Checking Packet Loss
Purpose Make sure that the communication between servers is normal.


i. Log in to each server as user "root" through Telnet or SSH.
#ping <gateway_IP>
¡ Windows
Select Start > Run. Enter cmd in the text box and click OK.
In the CMD window, run the following command:
ping <gateway_IP> -t
Standard for No request times out and no packets are lost for at least 20 min-
Reference utes.
Exception 1. Check whether the network cable is loose.
Handling
2. Check whether the plug of the network cable is in good condi-
tion.
3. Check switch logs.
4. Contact ZTE technical support engineers.
3.1.10Checking the Database Running

Status
Purpose Check the server of the database.

Instruction 1. Check database service ports.
� Oracle
Log in to the server as user "root" and run the following
command:
#netstat -an|grep 1521
Check whether the port is providing service.
� Sybase
command:
#netstat -an|grep 4100
Check whether the port is providing service.
� SQL Server
Select Start > Run. Enter services.mscin the text box
and then click OK to open the Services window. Check
whether the SQL Server(MSSQLSERVER) and SQL Server
Agent(MSSQLSERVER) services are in Started status.
2. Check processes.
� Oracle
command:
#ps -ef|grep ora_

The following key processes must be started:

– ora_reco_zxin
– ora_ckpt_zxin
– ora_smon_zxin
– ora_lgwr_zxin
– ora_pmon_zxin
– ora_dbw0_zxin
� Sybase
command:
#cd /opt/sybase/OCS-12_0/bin/
#isql -S<database name> -Usa -P<password>
1>sp_helpdb
2>go
View the database name list and run the following com-
mand:
1>sp_helpdb <database name>
2>go
� SQL Server
After you connect to the SQL Server database, select mas-
ter from the pull-down list box on the toolbar. Run the
Select * From master.sysprocessescommand to view the status
of the current SQL Server processes.
Standard for The database server processes are started.
Reference
The number of database connections is smaller than 80% of the
limit.
The space usage of the database is below 80%.
Exception If there are any exceptions, contact ZTE technical support engi-
Handling neers.
3.1.11Checking the Cluster
Purpose Make sure clusters run well.

mand:
¡ SUSE Linux
NewStart HA
#cli
cli:~>show -status
Check the status of the cluster. If the cluster runs normally,
the state of each node and heartbeat network is UP, and the
state of the service is running on at least one node.

¡ Windows
� Windows Cluster
Select Start > Programs > Tool Management > Cluster
Management. Select the default domain and go to the
management interface. If the cluster runs normally, all
resources are online on at least one server of the cluster.
Select Start > Programs > Tool Management > Cluster
Management. Select the default domain and go to the
resources are online on at least one server of the cluster.
� ZTE HA Cluster
Select Start > Programs > cluster manage to go to the
resources are displayed, the active device is highlighted in
grey and the standby device is green.
Reference
The state of each node and heartbeat network is UP, and the
state of the service is running on at least one node.
¡ Windows
� Windows Cluster
All resources are online on at least one server of the cluster.
� ZTE HA Cluster
All resources are displayed, the active device.
Exception Restart the cluster by running the following command:
Handling
¡ SUSE Linux
#cli
cli:~>cluster-start
3.1.12Checking WINNTTCP
Communication
Purpose Check whether the IMP server listens on the WINNTTCP port nor-
mally.
Run the netstat -an|grep port command on the IMP server to
display the listener port.
Or run the following command on the IMP server:
$imptool -c
TCP connection information is returned.
¡ Windows
Run the netstat -an|findstr port command on the IMP server
to display the listener port.

Standard for ¡ If the value of the noLimitPort field is 1 in the itcom.ini file,
Reference the port number configured through the client is used.
¡ If the value of the noLimitPort field is 0, port 5000 is used by
default.
Exception ¡ SUSE Linux
Handling
Log in as user "zxin10" and run the following command:
$impstat
Check the status of the IMP. If the impexe and impmoni pro-
cesses are not present or the output of the imptool -c com-
mand shows that the IMP fails to connect to the IMP, run the
$impstart monitor command to restart the process.
¡ Open Windows Task Manager by pressing Ctrl+Alt+Del. Click
the Processes tab. Check whether the impmonitor.exe and
imp.exe processes exist.
� If the impmonitor.exe or imp.exe process is not present,
open the IMP directory (C:\zxin10\imp by default), and
double-click impmonitor.exe to start the impmonitor
program (the monitoring program for the IMP). The imp-
monitor program starts the imp.exe process (main IMP
program) automatically.
� If the impmonitor.exe and imp.exe processes are
present, restart the IMP by performing the following steps:
a) Open Windows Task Manager by pressing
Ctrl+Alt+Del. Click the Processes tab.
b) Select impmonitor.exe and click End Process. In the
confirmation message box, click Yes.
c) Select imp.exe and click End Process. In the confirma-
tion message box, click Yes.
d) Open the IMP directory (C:\zxin10\imp by default),
and double-click impmonitor.exe to start the imp-
monitor program.
¡ Check whether ports are listened on. If not, contact ZTE tech-
nical support engineers.
3.1.13Checking the Connection

Between the Database Server
and the Database Client
Purpose Check whether the client of the database is connected to the server
of the database well.
Instruction ¡ Oracle
� If the client runs the Windows operating system, select
Start > Run on the client. Enter cmd in the text box and
click OK. In the CMD window, run the tnsping command
to ping the service name.

� If the client runs the SUSE Linux operating system, run the
$tnsping command in the command-line interface (CLI) to
ping the service name.
¡ Sybase
� If the client runs the Windows operating system, select
Start > Programs > Sybase > Dsedit Utility. Select
the default direction server to open the dsedit window.
Right-click Server and select Ping Server from the short-
cut menu. In the ping window, click ping. The output is as
shown in the figure below.
Select a server. Right-click Server Address and select

Ping Server from the shortcut menu.
� If the client runs the SUSE Linux operating system, run the
following command:
$isql -U<user_id> -P<password> -S<servername>
¡ SQL Server
� If the database is SQL Server 2000, select Start > Run.
Enter cmd in the text box and click OK. In the CMD window,
run the following command:
isql -U<user_id> -P<password> -S<servername>
Or
isql -U<user_id> -P<password> -S<server_ip>
� If the database is SQL Server 2005, select Start > Pro-
grams > Microsoft SQL Server 2005. Select SQL
Server Management Studio and log in as user "sa".
Standard for ¡ Oracle
Reference
In the output, the service name, host name, address and port
are displayed. The status is "OK"

¡ Sybase
i. In the dsedit window, right-click server and select Ping
Server from the shortcut menu. In the ping window, click
ping. The normal output is returned.
$isql -U<user_id> -P<password> -S<servername>
You can enter SQL Prompt.
¡ SQL Server
� SQL Server 2000
Run the following command:
isql -U<user_id> -P<password> -S<servername>
Or
isql -U<user_id> -P<password> -S<server_ip>
You can enter SQL Prompt.
� SQL Server 2005
You can enter SQL Server Management Studio. The icon
next to each server is green.
Exception ¡ Oracle
Handling
i. Run the following command to check the network between
the client of the database and the server:
ping <dbserver_ip>
ii. Check the $ORACLE_HOME/network/admin/tnsnames.ora
configuration file. Check whether the server name, IP
address, port, and protocol are configured correctly. For
details, refer to Oracle database installation manuals.
iii. Run the following command to check whether the server is
started normally:
$lsnrctl status
In the figure below, the database service name is zxin. The
output shows "no listener" and "connection refused". This
indicates that the database listener does not run normally.

Check the $ORACLE_HOME/network/admin/listener.ora

file. Run the following command to start the listener:
$lsnrctl start
iv. Log in to the database and check whether you can connect
to instances by running the following commands:
sqlplus /nolog
SQL>connect sys/<password>@<servername> as sysdba
The password is the login password of user "sys". "server-
name" stands for database instance name.
If the output shows "Connected", this indicates that the
database runs normally and is connected normally. If the
connection fails, check database parameters and restart
the database server. For details, refer to Oracle database
administration manuals.
v. For other faults, contact ZTE technical support engineers.
¡ Sybase
ping <dbserver_ip>
ii. Check the configuration file on the client side to see if the
server name, IP address, port, and protocol are configured
correctly. For details, refer to Sybase database administra-
tion manuals.
iii. Check whether the server is started normally.
a) Run the #ps -fusybase command. Check whether the
dataserver process is present in the output.
If the dataserver process is not present, this indicates
that the database server is not started. Check whether
the contents in the database configuration file are cor-
rect. Restart the database server. For details, refer to
Sybase database installation manuals.
b) Run the following command:
$isql -U<user_id> -P<password> -S<server_ip>
If SQL Prompt appears, this indicates that the server is
started and the client can connect to the server. If SQL
Prompt does not appear, contact ZTE technical support
engineers and restart the database server.

¡ SQL Server
ping <dbserver_ip>
ii. Check the configuration file on the client side to see if the
server name, IP address, port, and protocol are configured
correctly. For details, refer to SQL Server database admin-
istration manuals.
iii. On the server, select Start > Programs > Microsoft SQL
Server 2005. Select SQL Server Management Studio. If
you cannot log in as user "sa", restart the server by per-
forming the following steps:
a) Select Start > Programs > Microsoft SQL Server
2005 > Configuration Tools > SQL Server Config-
uration Manager.
b) In SQL Server Configuration Manager, expand Services
and click SQL Server (<instance name>).
c) In the detailed information pane, right-click the SQL
Server instance, and select Start.
d) If the fault persists, contact ZTE technical support en-
gineers.
3.1.14Checking IMP Processes
Purpose Check IMP processes.

i. Log in to the IMP server as user “zxin10”.
ii. Run the following command to check whether the impmoni
process is present:
$impstat
¡ Windows
Open Windows Task Manager by pressing Ctrl+Alt+Del. Click
the Processes tab. Check whether the impmonitor.exe and
imp.exe processes exist.
Reference
The impexe and impmoni processes are displayed.
¡ Windows
The imp.exe and impmonitor.exe processes are displayed.
Exception ¡ SUSE Linux
Handling
If the impexe or impmoni process is not present, log in to the
IMP server as user "zxin10" and run the following command to
start the impmoni and impexe processes:
$impstart monitor
¡ Windows

If the impmonitor.exe or imp.exe process is not present,

open the IMP directory (C:\zxin10\imp by default), and dou-
ble-click impmonitor.exe to start the impmonitor program
(the monitoring program for the IMP).
¡ Check whether both processes are started. If not, contact ZTE
technical support engineers.
3.1.15Checking the NFS Client
Purpose Check whether you can log in to the NFS server from the IMP.
Instruction 1. Check access to the NFS server from the NFS client.
Note:
� On the SUSE Linux operating system, the NFS client is rep-
resented by the NFS file system.
� On the Windows operating system, the NFS client is repre-
sented by a network mapped drive.
2. Check whether the NFS server is read normally.
Note:
� On the SUSE Linux operating system, enter the NFS file
system and check whether you can open its subdirectories
and copy one file to your local host.
� On the Windows operating system, enter the network
mapped drive and check whether you can open its subdi-
rectories normally and copy one file to your local host.
3. Check whether the NFS server is written normally.
Note:
� On the SUSE Linux operating system, enter the NFS file
system and copy one file from your local host to the NFS
server.
� On the Windows operating system, enter the network
mapped drive and copy one file from your local host to the
NFS server.
4. On the SUSE Linux operating system, run the df command to

check the connection to the NFS client.
In the figure below, "10.130.48.140:/home/vpopmail/do-
mains/" indicates the IP address and export directory of the
NFS server, and "/home/vpopmail/domains" indicates that
local mount point.

Standard for The NFS server can be read and written normally.
Reference
Exception 1. Check whether the network connection to the NFS server is
Handling normal by using the ping command.
2. On the Windows operating system, check whether the settings
of the network mapped drive are correct. If the username or
password is changed, map the drive again on the client. On
the SUSE Linux operating system, unmount the NFS file system
and mount it again.
3. If the fault persists, contact ZTE technical support engineers.
For the SUSE Linux operating system, send the /etc/fstabs
file to ZTE.
3.2 OMMP Checks

During weekly maintenance on the NGCSS, you need to perform

routine checks listed below on the OMMP.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
5 CPU usage rate 3.1.5
6 Memory usage rate 3.1.6
7 Swap area usage rate 3.1.7
8 Disk I/0 3.1.8
9 Network package loss 3.1.9
10 Dual-server status 3.1.11
WINNTTCP
11 3.2.2
communication status
Database connection
12 3.2.3
status

Daily Maintenance
No. Operation Guide
Item
13 Running status of task 3.2.4
14 OMMP process 3.2.5
3.2.2 Checking WINNTTCP

Communication
Purpose Make sure that the communication between the OMMP and other
modules is normal.
Instruction Log in to the OMMP as user "zxin10" and run the following com-
mand:
#zxtool -c
Check whether the connection between the OMMP and each other
module is normal.
Standard for The values in the PNode, Post and PeerIP columns are displayed
Reference normally and all the values in the Status column are 1, as shown
below.
Exception If the connection between the OMMP and other module is abnor-
Handling mal, check whether the network is normal and the module is al-
ready started.
3.2.3 Checking the Connection

Between the OMMP and the
Database
Purpose Make sure that the connection between the OMMP and the data-
base is normal.
Instruction 1. Log in to the OMMP as user "zxin10". Run the $zxstat com-
mand to check whether the sdfpro process is present.
2. Run the $netstat -an|grep 1521 command.
Standard for 1. In the output of the $zxstat command, the sdfpro process
Reference is present, as shown below. This indicates that the database
process is already started.

2. In the output of the $netstat -an|grep 1521 command, the

state is ESTABLISHED, as shown below. This indicates that
the system listens on port 1521 and the connection between the
server and the client is established successfully.
Exception If the connection between the OMMP and the database is abnor-
Handling mal, check whether the database is started normally and whether
the network between the OMMP and the database is faulty.
3.2.4 Checking Scheduled Tasks
Purpose Check every scheduled task of the OMMP.

mand:
$crontab -l
Standard for The scheduled tasks of the current user are displayed in the output
Reference of the $crontab -l command.
¡ In the Oracle database environment, the format of the output
(including time and command) is as follows:
0 0 1 * * mv /home/zxin10/task/result /home/zxin10/task/re-
sult.old 2>/dev/null
0 0 1 * * mv /home/zxin10/log/sdf.log
/home/zxin10/log/sdf.log.old 2>/dev/null
* * * * * /home/zxin10/bin/getsysinfo.sh >/dev/null 2>&1
0 8 * * * /home/zxin10/bin/gethwinfo.sh -f >/dev/null 2>&1
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
/home/zxin10/bin/checksyslog.sh >/dev/null 2>&1
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
/home/zxin10/bin/checkcluster.sh >/dev/null 2>&1
0 3 * * * /home/zxin10/bin/expdb.sh 1>/dev/null 2>/dev/null
0 6,18 * * * cp /home/zxin10/service/* /home/zxin10/serv-
ice_bak/
0 1 * * * /home/zxin10/bin/zxindbat chargetask.sql
0 5,11,17,23 * * * /home/zxin10/bin/zxindbat billtask.sql
0 2 * * * /home/zxin10/bin/zxindbat updatestat.sql
¡ In the Sybase database environment, the format of the output


sult.old
/home/zxin10/log/sdf.log.old
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
0 * * * * /home/zxin10/bin/zxindbat dtran.sql
0 3 * * * /home/zxin10/bin/zxindbat ddbak.sql
30 3 * * 0 /home/zxin10/bin/zxindbat dwbak.sql
ice_bak/
0 1 * * * /home/zxin10/bin/zxindbat chargetask.sql
0 5,11,17,23 * * * /home/zxin10/bin/zxindbat billtask.sql
0 2 * * * /home/zxin10/bin/zxindbat updatestat.sql
For details, refer to related engineering manuals.

Exception If there are any exceptions, check the logs of crontab (The path
Handling of logs is /home/zxin10/task/result) to see which tasks are not
executed. Contact ZTE technical support engineers.
3.2.5 Checking the OMMP Process
Purpose Check whether the OMMP process runs normally.

Instruction Log in to the OMMP as user "zxin10" and run the zxstat command.
Standard for If the smpser process is present, this indicates that the OMMP
Reference process runs normally. OMMP/Parlay Server processes are de-
scribed in the table below.
TABLE 1 OMMP/PARLAY SERVER PROCESSES
Parlay
Process Description OMMP
Server
zxmoni Monitoring process Yes Yes
zxinit Initialization process Yes Yes
zxcomm Communication process Yes Yes
System resource
R01MML Yes Yes
management process
sdfpro Database access process Yes Yes
Network management
snmpagt Yes Yes
process

Parlay
Process Description OMMP
Server
Parlay Server service

zxinserver No Yes
processing process
smpser OMMP management

Yes No
process
Synchronization proxy
ProxyPro Yes No
process
Exception If the smpser process is not displayed in the output of the zxstat
Handling command, restart the intelligent network or view the logs in the
zxcom.log file and handle accordingly.
3.3 SIP Proxy Server Checks

3.3.1 SIP Proxy Maintenance Task List

routine checks listed below on the SIP Proxy.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I/0 3.1.8
11 SIP Proxy process 3.3.2
3.3.2 Checking SIP Proxy Processes
Purpose Make sure that SIP proxy processes run normally.

Instruction 1. Log in to the SIP proxy server as user "zxin10" through Telnet
or SSH.
$zxstat
Standard for The following five processes are displayed in the output of the zx-
Reference stat command.
Exception If any of the five processes is not present, contact ZTE technical
Handling support engineers.

Checks
3.4.1 Oracle Database Server
Maintenance Task List

routine checks listed below on the Oracle database server.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I/0 3.1.8
Connection status of
11 3.1.13
database client

Daily Maintenance
No. Operation Guide
Item
Automatic backup
12 3.4.2
function of database
Resources occupied by
13 3.4.3
database process
14 Running status of JOB 3.11.4
3.4.2 Checking Automatic Database

Backup
Purpose Check whether daily and weekly backup tasks for the database run
normally.
Instruction 1. Log in to the database server as user "zxin10" and run the
following command:
$crontab -l
The database backup tasks are displayed in the output of the
$crontab -l command. The format is as follows:
� Oracle
� Sybase
2. In the Oracle database environment, check the time and size of

the dmp data backup file in the /zxindata/zxinbak directory
(on SUSE Linux) or in the /zxinbak directory (on Linux).
In the Sybase database environment, check the time and size
of the zxinsys_dbak, master_wbak, zxin_bak, and zxin-
sys_wbak files in the /sybak directory.
Standard for In the Oracle database environment, new files are generated in
Reference the /zxindata directory. The output of the $crontab -l command
shows that the expdb.sh script is run at scheduled times.
In the Sybase database environment, a new zxinsys_dbak file
is generated every day and new master_wbak, zxin_bak, and
zxinsys_wbak files are generated every week in the /sybak di-
rectory.

3.4.3 Checking the Resource Usage of

Database Processes
Purpose Check the memory usage and CPU usage of database processes.
Instruction 1. Log in to the database server as user “root” and run the top
command.
2. Observe and record the process name (COMMAND), CPU usage
(CPU%), process ID (PID), and memory usage (RES) of each
process when the system is busy.
Standard for The CPU usage of each process cannot be greater than 30% for
Reference a long time or greater than 60% for more than 5 seconds. The
memory usage keeps rising.
Exception If there are any exceptions, record the output and contact ZTE
Handling technical support engineers.
3.4.4 Checking Database Jobs
Purpose Check database jobs.

Instruction Take the Oracle database for example. Log in to the database as
user "sys" on the Oracle client and run the following SQL state-
ment:
select * from dba_jobs
Check the BROKEN and FAILURES fields in the output.
Standard for All the values in the BROKEN field are N, and all the values in the
Reference FAILURES field is 0.
Exception If there are any exceptions, record the output and contact ZTE
3.5 Parlay Server Checks

3.5.1 Parlay Server Maintenance Task
List

routine checks listed below on the Parlay Server.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2

Daily Maintenance
No. Operation Guide
Item
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I0 3.1.8
WINNTTCP
11 3.5.2
Database connection
12 3.5.3
status
13 Running status of task 3.5.4
14 Parlay Server process 3.5.5

Communication
Purpose Make sure that the communication between the Parlay Server and
other modules is normal.
Instruction Log in to the Parlay Server as user "zxin10" and run the following
command:
$zxtool -c
Check whether the connection between the Parlay Server and each
other module is normal.
Standard for The values in the PNode, Post and PeerIP columns are displayed
Reference normally and all the values in the Status column are 1, as shown
below.
Exception If the connection between the Parlay Server and other module is
Handling abnormal, check whether the network is normal and the module
is already started.

3.5.3 Checking the Connection

Between the Parlay Server and
the Database
Purpose Make sure that the connection between the Parlay Server and the
database is normal.
Instruction 1. Log in to the Parlay Server as user "zxin10". Run the $zxstat
command to check whether the sdfpro process is present.
2. Run the $netstat -an|grep 1521 command.
Standard for 1. In the output of the $zxstat command, the sdfpro process
Reference is present, as shown below. This indicates that the database
process is already started.
2. In the output of the $netstat -an|grep 1521 command, the

state is ESTABLISHED, as shown below. This indicates that
the system listens on port 1521 and the connection between the
server and the client is established successfully.
Exception If the connection between the Parlay Server and the database is
Handling abnormal, check whether the database is started normally and
whether the network between the Parlay Server and the database
is faulty.
3.5.4 Checking Scheduled Tasks
Purpose Check every scheduled task of the Parlay Server.

command:
$crontab -l
Standard for The scheduled tasks of the current user are displayed in the output
Reference of the crontab –lcommand.
¡ In the Oracle database environment, the format of the output

sult.old 2>/dev/null
/home/zxin10/log/sdf.log.old 2>/dev/null
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
ice_bak/
¡ In the Sybase database environment, the format of the output

sult.old
/home/zxin10/log/sdf.log.old
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
ice_bak/
For details, refer to related engineering manuals.

3.5.5 Checking the Parlay Server

Process
Purpose Check whether the Parlay Server process runs normally.

Instruction Log in to the Parlay Server as user "zxin10" and run the $zxstat
command.
Standard for If the zxinserver process is present, this indicates that the Parlay
Reference Server process runs normally.
Exception Restart the intelligent network or view the logs in the zxcom.log
Handling file and handle accordingly.

3.6 EBASE Server Checks

3.6.1 EBASE Server Maintenance
Task List

routine checks listed below on the EBASE server.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I0 3.1.8
11 EBASE process 3.6.2
3.6.2 Checking the EBASE Processes
Purpose Make sure that EBASE processes run normally.

Instruction 1. Log in to the EBASE server as user "xbase" through Telnet or
SSH.
$xstat
Standard for The xbasemoni and extrabase processes are displayed in the
Reference output of the $xstat command.
Exception 1. In a single-server configuration, start the xbasemoni process.

Handling
2. In a two-server cluster configuration, check whether cluster
resources are online. If not, bring cluster resources online.

3.7 Smart Call Alert Checks

3.7.1 Smart Call Alert Maintenance
Task List

routine checks listed below on the Smart Call Alert.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I0 3.1.8
WINNTTCP
11 3.7.2
Database connection
12 3.1.13
status
Smart Call Alert

13 3.7.3
process

Communication
Purpose Check whether the Smart Call Alert is connected to the Parlay
Server and the VGW well.
Instruction Run the $netstat -an|grep 5000 command on the Smart Call Alert
server to check the connections between the Smart Call Alert and
the Parlay Server and the VGW.

Standard for The normal output is as shown in the figure below. The
Reference client (10.130.56.65) is connected to port 5000 of the server
(10.130.56.51).
zxin10@In_SmsPro1:~> netstat -an|grep 5000

tcp 0 0 10.130.56.51:5000 0.0.0.0:* LISTEN
tcp 0 0 10.130.56.65:47606 10.130.56.51:5000 ESTABLISHED
Exception Enter ./s10dog & in the /home/zxin10/smtrans/bin directory to

Handling start the watchdog. The watchdog starts the smtransf process.
3.7.3 Checking Smart Call Alert

Service Processes
Purpose Check whether Smart Call Alert service processes are started nor-
mally.
Instruction Telnet to the Smart Call Alert server as user "zxin10" and run the
$ps -ef|grep smtransf command.
Standard for The smtransf process is present in the output of the $ps -ef|grep
Reference smtransf command.
Exception If the smtransf process is not present, enter ./s10dog & in the
Handling /home/zxin10/smtrans/bin directory to start the watchdog. The
watchdog starts the smtransf process.
3.8 VGW Checks

3.8.1 VGW Maintenance Task List

routine checks listed below on the VGW.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I0 3.1.8

Daily Maintenance
No. Operation Guide
Item
WINNTTCP
11 3.8.2
12 VGW process 3.8.3

Communication
Purpose Check whether the VGW is connected to the Parlay

Server/SMSC/SMS gateway well.
Instruction Run the $netstat -an|grep <port> command on the VGW server.
The port number of the Parlay Server is 5000. The port number
of the SMSC/SMS gateway is indicated in the /home/zxin10/sm-
sagent/config/zxvms.ini file, as shown below.
smscIPAddr =10.130.110.166; //IP address of the SMSC/SMS gateway

SmscPort =5016
Exception Check whether the WINNTTCP service is enabled on the VGW

Handling server.
Enter ./s10dog & in the /home/zxin10/smsagent/bin directory.
3.8.3 Checking the VGW Process
Purpose Make sure that one and only one VGW process is present.
Instruction Telnet to the VGW server as user "root" and run the $ps -ef|grep
smsagent_smpp command. The following information is displayed:
root 28630 28397 0 Apr01 ? 00:02:23 ./smsagent_smpp
Exception If the VGW process is not started, enter ./s10dog & in the
Handling /home/zxin10/smsagent/bin directory to start the watchdog.
The watchdog starts the smsagent_smpp process.

3.9 Media Server (CMS9000/3000)

Checks
3.9.1 Media Server Maintenance Task
List

routine checks listed below on the media server.
Daily Maintenance
No. Operation Guide
Item
Network bandwidth
2 3.9.3
usage of MPC board
3.9.2 Checking the CPU Usage of

Media Server
Purpose Make sure that the CPU usage of the SCC/MPC board is normal.
"xxx.xxx.xxx.xxx" indicates the maintenance address of the
SCC board.
3. Select Performance Mgt > Show Real-Time Statistics.
4. Select the active SCC board (slot 1 or slot 14) and click Exe-
cute.

5. For the MPC board, select slot 2 to slot 13 and click Execute.
Standard for The value of the Max CPU Utilization field is not greater than
Reference 70%. The value of the Avg CPU Utilization field is not greater
than 30%.
Exception Select Performance Mgt > Retrieve Statistics. Select Web
Handling Download. Click the Save Settings button. The Download
statistics file link is displayed. Click this link and save the Sta-

tistics.txt file to your local host. Send this file to ZTE technical
support engineers.
3.9.3 Checking the Bandwidth Usage

of the MPC Board
Purpose Make sure that the network read/write of the CMS9000/3000 is

normal.
"xxx.xxx.xxx.xxx" indicates the address of the maintenance
console of the SCC board.
3. Select Performance Mgt > Show Real-Time Statistics.
4. For the MPC board, select slot 2 to slot 13 and click Execute.
Standard for The value of the Avg Bandwidth Utilization field is below 50%
Reference and the value of the Max Bandwidth Utilization field is below
80%.
Exception Select Performance Mgt > Retrieve Statistics. Select Web
Handling Download. Click the Save Settings button. The Download sta-
tistics file link is displayed. Click this link and save the Statistics.txt
file to your local host. Send this file to ZTE technical support en-
gineers.

3.10 Network Storage Checks

3.10.1Network Storage Maintenance
Task List

routine checks listed below on the Network Storage.
Daily Maintenance
No. Operation Guide
Item
1 I node 3.10.2
2 Disk space 3.10.3
3.10.2Checking Inodes
Purpose Make sure that there are sufficient idle inodes.

Instruction Log in to the server as user "root" and run the following command:
>df -i
Standard for The value of the %iused field is below 75%.
Reference
Exception If the value of the %iused field is above 75%, contact ZTE tech-
Handling nical support engineers.
3.10.3Checking Disk Space of Network

Storage
Purpose Make sure that there is sufficient idle disk space.

Instruction Log in to the server as user "root" and run the following command:
>df
Standard for The value of the capacity field is below 75%.
Reference
Exception If the value of the capacity field is above 75%, contact ZTE tech-
Handling nical support engineers.

3.11 Web Server Checks

3.11.1Web Server Maintenance Task
List

routine checks listed below on the Web server.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I0 3.1.8
Database connection
11 3.1.13
status
12 Web-based login 3.11.2
13 Tomcat process 3.11.3
14 JBOSS process 3.11.4
3.11.2Checking Web Login
Purpose Make sure that the Web server works normally and registration
and deregistration in the BOSS system are normal.
Instruction Check whether you can open the Web pages for the VMS and the
MCA as user "admin" through IE.
Standard for Web pages are displayed normally. Operations on Web pages are
Reference accomplished normally.
Exception 1. Log in to the Web server as user "root" through Telnet or SSH
Handling and run the following command:
$ps -ef|grep jboss

If the jboss process is not present, restart the Web service

and contact ZTE technical support engineers.
2. Check VMS log files.
Tomcat logs: /home/zxin10/was/tomcat/logs/catalina.out
JBoss logs: /usr/java/jboss/server/default/log
3. Check MCA log files.
/usr/java/zxmc/tomcat-5.0.28/logs/catalina.out
/usr/java/zxmc/jboss/server/default/log
If logs contain error messages such as "error" and "failed",
contact ZTE technical support engineers.
3.11.3Checking Tomcat Processes
Purpose Make sure that Tomcat processes are started normally.

Instruction 1. Log in to the Web server as user "root".
#ps -ef|grep tomcat
Standard for The following processes exist.
Reference
root 5606 1 0 Jun20 ? 00:04:12 /opt/jdk1.5.0_07/jre/bin/java
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=/opt/tomcat-5.5.25/conf/logging.prop-
erties
-Djava.endorsed.dirs=/opt/tomcat-5.5.25/common/endorsed
-classpath :/opt/tomcat-5.5.25/bin/bootstrap.jar:/opt/tomcat-
5.5.25/bin/commons-logging-
api.jar
-Dcatalina.base=/opt/tomcat-5.5.25 -Dcatalina.home=/opt/tomcat-
5.5.25
-Djava.io.tmpdir=/opt/tomcat-5.5.25/temp org.apache.cata-
lina.startup.Bootstrap start
root 10232 1 0 Jun20 ? 00:00:08 /opt/NewStar-
tHA/web/jdk1.6.0_01/bin/java
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=/opt/NewStartHA/web/tomcat/conf/log-
ging.properties
-Djava.endorsed.dirs=/opt/NewStartHA/web/tomcat/common/en-
dorsed
-classpath :/opt/NewStartHA/web/tomcat/bin/boot-
strap.jar:/opt/NewStartHA/web/tomcat/
bin/commons-log
ging-api.jar -Dcatalina.base=/opt/NewStartHA/web/tomcat
-Dcatalina.home=/opt/New-
StartHA/web/tomcat
-Djava.io.tmpdir=/opt/NewStartHA/web/tomcat/temp
org.apache.catalina.startup.Bootstrap start

Exception If any process is not present, run the following commands:

Handling
#cd /home/zxin10/was
#sh apache.sh start
#sh tomcat.sh start
#cd /usr/java/zxmc/tomcat-5.0.28/bin
#sh startup.sh &
3.11.4Checking JBoss Processes
Purpose Make sure that JBoss processes are started normally.

Instruction 1. Log in to the Web server as user "root".
#ps -ef|grep jboss
Standard for The following processes exist.
Reference
root 8106 8101 0 Dec02 pts/6 00:05:44
/usr/java/j2sdk1.4.2_10/bin/java -server -Xms256m -Xmx512m
-Dprogram.name=run.sh -classpath /usr/java/Jboss-
3.2.3/bin/run.jar:/usr/java/j2sdk1.4.2_10/lib/tools.jar org.jboss.Main
root 8275 8050 0 Dec02 pts/7 00:03:41
/usr/java/j2sdk1.4.2_10/bin/java -server -Xmx512m
-Xms256m -Djava.awt.headless=true -Djava.en-
dorsed.dirs=/usr/java/Tomcat-5.0.28/common/endorsed
-classpath /usr/java/j2sdk1.4.2_10/lib/tools.jar:/usr/java/Tom-
cat-5.0.28/bin/bootstrap.jar:/usr/java/Tomcat-5.0.28/bin/com-
mons-logging-api.jar -Dcatalina.base=/usr/java/Tom-
cat-5.0.28 -Dcatalina.home=/usr/java/Tomcat-5.0.28
-Djava.io.tmpdir=/usr/java/Tomcat-5.0.28/temp -Djava.nam-
ing.factory.initial=org.jnp.interfaces.NamingContextFactory
-Djava.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
-Djava.naming.provider.url=jnp://127.0.0.1:1099 -Djava.awt.head-
less=true org.apache.catalina.startup.Bootstrap start
Exception If any process is not present, run the following commands:

Handling
#cd /usr/java/jboss/bin
#sh run.sh &
#cd /usr/java/zxmc/jboss-3.2.3/bin
#sh run.sh &
3.12 Notify Scheduler Check

routine checks listed below on the Notify Scheduler.

Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I0 3.1.8
WINNTTCP
11 3.1.12
Database connection
12 3.1.13
status
13 IMP process 3.1.14
14 NFS client status 3.1.15
3.13 UMA Checks

3.13.1UMA Maintenance Task List

routine checks listed below on the UMA.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4

Daily Maintenance
No. Operation Guide
Item
8 Disk I0 3.1.8
Database connection
11 3.1.13
status
12 UMA process 3.13.2
3.13.2Checking UMA Processes
Purpose Check whether UMA processes run normally.

Instruction ¡ To check the UMA service processor processes, run the follow-
ing commands:
#su -u zxuma
$zxstat
¡ To check the UMA management portal processes, run the fol-
lowing commands:
#su -u zxuma
$ps -ef|grep jboss
Standard for ¡ UMA service processor processes
Reference
The ps command can show the status of the current processes,
as shown below. The following processes must be present:
zxmoni, zxinit, zxcomm, zxuep, smpp, and sdfpro.
$zxstat
zxuma 1791 1 7 10:52 ? 00:00:00 zxmoni
zxuma 1432 1 2 10:52 ? 00:00:00 zxinit
zxuma 1679 1 0 10:52 ? 00:00:00 zxcomm
zxuma 1700 1 3 10:52 ? 00:00:00 zxuep
zxuma 1756 1 6 10:52 ? 00:00:00 smpp
zxuma 1768 1 1 10:52 ? 00:00:00 sdfpro
¡ UMA management portal processes

The ps command can show the status of the current processes,
as shown below. The jboss and tomcat processes must be
present.

Exception ¡ If any UMA service processor process is not present, perform

Handling the following steps:
� If all UMA processes are stopped, restart UMA processes
by switching the cluster and contact ZTE technical support
engineers.
� If UMA processes are restarted abnormally for several
times, check the network connection by using the ping
command. If the network connection is broken or a large
number of requests time out, contact network engineers.
� If the sdfpro process is abnormal, a large number of core
files would be generated. Check whether the database
runs normally. For the detailed check procedure, refer
to the "3.1.13" section. If the database is disconnected,
check whether the database service and the database client
process run normally. If you cannot recover the connection
to the database, contact ZTE technical support engineers.
� Check the /home/zxuma/log/zxcom.log system log file to
see if there are any records containing "errors" or "fail". If
there are error logs, back up logs and contact ZTE technical
support engineers.
¡ Troubleshooting UMA management portal processes
If the JBoss and Tomcat processes are not started, perform the
following steps:
i. Telnet to the UMA server.
#su - zxuma
$cd /usr/umaweb
$./jboss.sh
$./tomcat.sh run
ii. If the JBoss or Tomcat process cannot be started normally,
check the Tomcat log file /usr/umaweb/tomcat/logs/zx-
uma.log. Send this log file to ZTE technical support engi-
neers.


3.14.1Mail Processor Maintenance
Task List

routine checks listed below on the Mail Processor.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I0 3.1.8
WINNTTCP
11 3.14.2
12 Mail Processor process 3.14.3
13 Mail Processor and 3.14.4
Network Storage
3.14.2Checking WINNTTCP
Communication
Purpose Check whether the WINNTTCP communication port of the Mail Pro-
cessor is normal.
Instruction 1. Log in to the server as user "root".
#netstat -an|grep port Check whether TCP monitors port
5000 and the status of port 5000 is "ESTABLISHED".
Standard for If the following output is displayed, this indicates that the WIN-
Reference NTTCP protocol is enabled on 10.130.57.142 and the connection
between 10.130.57.142 and 10.130.48.53 is established.

Exception ¡ If the connection is not established, perform the following

Handling steps:
i. Log in to the Mail Processor as user "root".
ii. Check whether the /usr/zte/mdss/etc/win_mgt.ini file
is correct.
iii. Save the /usr/zte/mdss/logs/zxcom.log file and send it
to ZTE technical support engineers.
¡ Restart the Mail Processor process with the assistance of ZTE
technical support engineers.
i. Run the #cd /usr/zte/mdss/bin command.
ii. Run the #./superstart_MDSS command.
3.14.3Checking Mail Processor

Processes
Purpose Make sure that the Mail Processor processes work normally.
Instruction 1. Log in to the Mail Processor as user "root".
#ps -ef | grep MDSS
Standard for The following processes are present in the output of the command.
Reference
Exception ¡ Restart the Mail Processor processes.

Handling
i. Log in to the Mail Processor as user "root".
ii. Run the following commands:
#cd /usr/zte/mdss/bin
#./superstart_MDSS
¡ Contact ZTE technical support engineers.
3.14.4Checking the Connection

Between the Mail Processor and
the Network Storage
Purpose Make sure that the connection between the Mail Processor and the
Network Storage is normal.
Instruction 1. Log in to the Mail Processor server as user "root".
2. Check whether the NFS is mounted to the Mail Processor. Sup-
pose that the address of the Network Storage is 10.130.44.215,

the shared directory is /vol/voldata1, and the mount point

on the MRC is /netappnfs.
i. In the /etc/fstab file, view the information about the
mounted NFS file.
10.130.44.215:/vol/voldata1 /netappnfs nfs

rsize=8192,wsize=8192,soft,timeo=2, intr 0 0

#mount
Check whether the output contains a character string in the
following format:
10.130.44.215:/vol/voldata1 on /netappnfs type nfs

(rw,intr,bg,rsize=8192,wsize=8192,lock,timeo=300,hard,add-
r=10.130.44.215)
3. Read the files on the Network Storage from the Mail Processor
by running the following command:
#cd /netappnfs
#ls
Standard for 1. In the output of the mount command, the actual mount point of
Reference the NFS is consistent with the configuration in the /etc/fstab
file.
2. The contents of the cd /netappnfs directory are listed in the
output of the ls command.
Exception 1. Run the following command:
Handling
#ping <Network Storage_IP>
Check the network connection between the client and the
server.
2. Check whether the NFS service on the Network Storage is nor-
mal.
3.15 Mail Clear Server Check

routine checks listed below on the Mail Clear.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4

Daily Maintenance
No. Operation Guide
Item
8 Disk I0 3.1.8
WINNTTCP
11 3.1.12
Database connection
12 3.1.13
status

routine checks listed below on the statistics server.
Daily Maintenance
No. Operation Guide
Item
1 NIC status 3.1.1
2 System time 3.1.2
3 Core file 3.1.3
4 Disk space 3.1.4
8 Disk I0 3.1.8
WINNTTCP
11 3.1.12
Database connection
12 3.1.13
status

Daily Maintenance
No. Operation Guide
Item
3.17 Checking Power Supplies of

Firewall
Purpose Check whether the power supplies of the firewall are normal.
Instruction Check if there is any abnormal heating or smells.
Standard for No abnormal heating or smells.
Reference
Exception Check power cable connectors and other components. Check the
Handling voltage and power. Take actions if there is any problem.


�4 Monthly Maintenance
>> Universal Monthly Maintenance Items
>> OMMP Checks
>> Parlay Server Checks
>> SIP Proxy Server Checks
>> EBASE Server Checks
>> Smart Call Alert Checks
>> VGW Checks
>> Oracle Database Server Checks
>> Web Server Checks
>> UMA Checks
>> Mail Clear Server Checks
>> Sybase Database Server Checks
>> Firewall Checks
>> Layer 3 Switch and GRE Router Checks
4.1 Universal Monthly

Maintenance Items
4.1.1 Checking Power Supplies
Purpose Check the power modules of all devices.

Instruction Check whether there is any abnormal heating or smell. Check
whether any indicator on the instrument panel of the power dis-
tribution cabinet is red.
Standard for There is no abnormal heating or smell.
Reference
Both power supplies of PC servers and servers work normally.
Exception If any power supply is faulty, contact ZTE technical support engi-
Handling neers and replace power devices.

4.1.2 Checking Operating System

Patches
Purpose Check the version and patches of the operating system.

Log in to the operating system and run the #uname -a com-
mand.
¡ Windows
Right-click My Computer and select Properties to view the
patches of the Windows operating system. In the Control
Panel window, select Add or Remove Programs to check
the situations of these patches.
Standard for The patches required for the operating system version have been
Reference installed.
Exception If there are exceptions, contact ZTE technical support engineers.
Handling
4.1.3 Freeing Up Disk Space
Purpose Delete expired log files and other large files to ensure that there
is sufficient disk space.
Instruction Delete files with the assistance of ZTE technical support engineers.
1. To delete large files, perform the following steps:
� SUSE Linux
Search for large files by running the following command:
#find / -size +200000 -exec ls -l {} \; > a.txt
This command searches for the files whose size is large
than 200000 blocks (512 bytes per block) and lists these
files in the a.txt file.
Send the a.txt file to ZTE technical support engineers.
Delete useless files and expired backup files with the as-
sistance of ZTE technical support engineers.
� Windows
Select Start > Search > All Files and Folders.
Enter * in the All or part of the file name text box. Select
My Computer from the Look in pull-down list box. Click
What size is it?, select Specific size, select at least, and
enter 500000. Click Search. The system searches for the
files whose size is greater than 500000 KB.
Send search results to ZTE technical support engineers.
Delete useless files and expired backup files with the as-
sistance of ZTE technical support engineers.
2. To delete log files, perform the following steps:
� SUSE Linux

Chapter 4 Monthly Maintenance
SUSE Linux system log files are stored in the /var/log

directory. Warning and error messages are stored in the
/var/log/warn file. Other logs are mainly stored in the
/var/log/messages file. Log files are managed through
the syslog daemon. The message buffer of the kernel is
viewed by the #dmesg command.
File Program Source Frequency Version Description
Logs of the
apache2/* httpd V2 F D S Apache HTTP
server V2
Output of
boot.log RC script Fc M R the system
startup script
cron cron S W R Crontab logs
Dumped
boot.msg kernel H S kernel
messages
cups/* CUPS F W All Print-related
Unsuccessful
faillog login H W S login
attempts
Last login
lastlog login H All time of each
user (binary)
Mail function
mail* Mail-related S W All
messages
messages many Main system

S W All
log files
All warning
warn many S W S and error
messages
Login records
wtmp login H M All
(binary)
X window
Xrog.n.log Xorg F W RS
server errors
Source: S=syslog, H=hard link, F=configuration file

Frequency: D=daily, W= weekly, M=monthly
Version: R=Red Hat Enterprise Linux, S=SUSE
The passwd, login, shutdown records are written to the au-
thorization logs. For Red Hat Enterprise Linux, the storage
path is /var/adm.
The /var/log/wtmp file stores user login and logout records
in binary. View these logs by using the last command.

The /var/log/lastlog file records the last login time of

each user. The size of this log file keeps unchanged unless
new users are added.
� Windows
a) Select Start > Settings > Control Panel > Admin-
istrative Tools > Event Viewer.
b) Under Event Viewer (Local), right-click Application
and select Clear all Events. A dialog box is displayed.
Click Yes to save logs before deletion. Set the log file
type to TXT, specify a name for the log file, and click
Save.
c) Repeat the same step to save and remove system logs
and security logs.
Standard for ¡ On the Linux operating system, the space usage of the /usr
Reference and /opt file systems is below 90%, and the space usage of
any other file system is below 80%.
¡ On the Windows operating system, the free space of the root
partition (normally, drive C) is greater than 2 GB. The space
usage of any other partition is below 80%.
Exception If there are any exceptions, contact ZTE technical support engi-
Handling neers.
4.1.4 Checking Operating System

Logs
Purpose Check system logs to see if any error is reported.

mand:
¡ SUSE Linux
#more /var/log/messages
¡ Windows
Select Start > Control Panel > Administrative Tools >
Event Viewer. Check application program logs, system logs
and security logs.
Reference
The /var/log/messages file does not contain error records
such as "Error", "WARNING", "FAILED", "failure", "Closing",
"down", "not responding" and "missing".
¡ Windows
Logs do not contain Error and WARNING records. The Type
column does not contain the and icons.

Exception Back up logs and contact ZTE technical support engineers.

Handling
4.1.5 Checking Cluster Logs
Purpose Check if cluster status is normal.

Instruction Check cluster logs with the assistance of ZTE technical support
engineers.
To check cluster logs, perform the following steps:
¡ SUSE Linux
� NewStartHA
The logs of a NewStartHA cluster are stored in the
/var/log/ha-log file.
� Skybility HA
Skybility HA Service writes logs using syslog. The default
path is /var/log/ha.
Skybility HA Service supports the log rotation feature.
When the size of the logs reaches the limit, the system
compresses, transfers and deletes original logs automat-
ically. For example, if the size of the /var/log/ha file
reaches 4 MB, ha is compressed to ha.1, the original ha.1
is changed to ha.2 and the original ha.2 is changed to
ha.3 and so on. The oldest is ha.99. The storage duration
of logs is one year. When the storage duration of a log
file reaches one year, this file is deleted. When ha.98 is
changed to ha.99, the original ha.99 is deleted.
¡ Windows
On the Windows operating system, cluster logs are stored in
system logs. Refer to the "4.1.4" section.
Standard for Log files do not contain error records such as "Error", "WARNING"
Reference and "Failed".
Exception Back up part or all of the logs. Handle the fault based on the logs.
Handling Contact ZTE technical support engineers.
4.1.6 Checking Database Logs
Purpose Find log errors in the database and maintain database logs.
Instruction 1. Check errors.
� Oracle
The Oracle database logs are stored in the $ORA-
CLE_BASE/admin/<SID>/bdmp/alert_<SID>.log file. "SID"
indicates the database instance name. $ORACLE_BASE
is an environment variable, and the value is normally
/home/oracle. The alert_<SID>.log file stores logs of
system startup, shutdown, errors, storage change and log
switch in text.

The $ORACLE_BASE/admin/<SID>/bdmp directory stores er-

ror log files for background processes by process name and
process number. The extension is trc. The text format is
specified by the background_dump_dest parameter.
Core dumps of Oracle service processes are stored in the
$ORACLE_BASE/admin/<SID>/cdump directory. The text for-
mat is specified by the core_dump_dest parameter.
User trace files store user process error logs and use trc
as the extension. These files are stored in the $ORA-
CLE_BASE/admin/<SID>/udump directory. The text format
is specified by the user_dump_dest parameter.
� Sybase
Sybase database log files are stored in the $HOME/ASE-<ver-
sion>_install directory. For the version part, 12_0 stands
for 12.0, 12_5 stands for 12.5, and 15_0 stands for 15.0.
The log file is named "sybase_<servername>.log", and the
backup log file is named "sybase_<servername>_backup.log".
"servername" stands for the database name.
� SQL Server
SQL Server writes certain system events and user-defined
events to SQL Server error logs and Windows application
logs.
Windows application logs records events on the Windows
operating system, SQL Server and SQL Server Agent.
SQL Server error logs may be viewed through SQL Server
Management Studio or any text editor. By default, SQL
Server error logs are stored in the ERRORLOG and ER-
RORLOG.n files in the Program Files\Microsoft SQL
Server\MSSQL.n\MSSQL\LOG directory. Normally, SQL
Server stores the last six log files. The current log file is
ERRORLOG. The last log file is ERRORLOG.1, the last but one
log file is ERRORLOG.2 and so on.
2. Delete expired logs.
� Oracle
a) Run the ftp or cp command to back up the files in
the $ORACLE_BASE/admin/<SID> directory. For exam-
ple, to back up all the logs of the zxin database to
the /home/oracle/backup directory, run the following
commands:
$cp -r /home/oracle/admin/zxin/bdump /home/oracle/backup
$cp -r /home/oracle/admin/zxin/udump /home/oracle/backup
$cp -r /home/oracle/admin/zxin/cdump /home/oracle/backup
Note:
On the Windows operating system, use the copy com-
mand.
b) Run the following commands to delete alert log files and

*.trc log files:

$rm /home/oracle/admin/<sid>/bdmp/alert_<sid>.log
$rm /home/oracle/admin/<sid>/bdmp/*.trc
Example:
To delete alert log files and *.trc log files of the zxin
database, run the following command:
$rm /home/oracle/admin/zxin/bdmp/alert_zxin.log
$rm /home/oracle/admin/zxin/bdmp/*.trc
Note:
On the Windows operating system, use the delete com-
mand.
c) Run the following commands to delete core dumps and

user process error logs:
$rm /home/oracle/admin/<sid>/cdmp/*.trc
$rm /home/oracle/admin/<sid>/udmp/*.trc
Example:
To delete core dumps and user process error logs of the
zxin database, run the following commands:
$rm /home/oracle/admin/zxin/cdmp/*.trc
$rm /home/oracle/admin/zxin/udmp/*.trc
Note:
On the Windows operating system, use the deletecom-
mand.
� Sybase
a) Back up database logs.
Run the ftp or copy command to back up the $ASE_<ver-
sion>/install/sybase_<servername>.log and $ASE_<ver-
sion>/install/sybase_<servername>_backup.log files.
For example, to back up all the logs of the zxin data-
base of Sybase 12.5.X to the /home/sybase/backup
directory, run the following commands:
$cp /home/sybase/ASE_12-5/install/sybase_zxin.log
/home/sybase/backup
$cp /home/sybase/ASE_12-5/install/sybase_zxin_backup.log
/home/sybase/backup

Note:
mand.
b) Delete log files.

$rm /home/sybase/ASE_12-5/install/sybase_zxin_backup.log
$rm /home/sybase/ASE_12-5/install/sybase_zxin.log
Note:
mand.
� SQL Server
a) SQL Server Management Studio
Back up and delete ERRORLOG and ERRORLOG.n
files in the Program Files\Microsoft SQL
Server\MSSQL.n\MSSQL\LOG directory.
b) Back up and delete system logs.
Select Start > Settings > Control Panel > Adminis-
trative Tools > Event Viewer. Under Event Viewer
(Local), right-click Application and select Clear all
Events. A dialog box is displayed. Click Yes to save
logs before deletion. Set the log file type to TXT, specify
a name for the log file, and click Save.
Standard for ¡ Logs are backed up before deletion.
Reference
¡ Logs of the last 60 days exist.
¡ Database log files do not contain error records such as "Error"
and "Warning".
Exception ¡ If logs are not maintained for a long time and causes insuffi-
Handling cient storage space and consequently database errors, delete
logs directly.
¡ If logs are not maintained for a long time, deleting many logs
at a time may cause errors. Delete logs in batches with the
assistance of ZTE technical support engineers.
4.1.7 Checking IMP Logs
Purpose Make sure that the IMP runs normally.

Instruction IMP log files include imptrace.dat, imptrace_x.dat, and ser-
vicesxxx.log. "xxx" stands for the IMP interface code.
¡ On the Windows operating system, log files are in the
c:\zxin10\imp\log directory.

¡ On the SUSE Linux operating system, IMP V1.x log files are
in the /home/zxin10/impsys/log directory, and IMP V2.x log
files are in the /home/zxin10/log directory. Log in to the IMP
server as user "zxin10" by using the FTP tool and download log
files to the client to view.
Standard for Log files do not contain error records such as "error" and "failed".
Reference
Exception Send the downloaded log files to ZTE technical support engineers.
Handling
4.2 OMMP Checks

During monthly maintenance on the NGCSS, you need to perform

routine checks listed below on the OMMP.
Daily Maintenance
No. Operation Guide
Item
1 Power supply to server 4.1.1
2 OS patch 4.1.2
3 Release of disk space 4.1.3
4 OS log 4.1.4
Application log of
5 4.2.2
OMMP
6 Dual-server log 4.1.5
Backup of application
7 4.2.3
program and log file
4.2.2 Checking OMMP Logs
Purpose Discover potential problems of the OMMP by checking OMMP logs.

Instruction 1. Log in to the OMMP as user "zxin10" by using the FTP tool.
2. Download the sdf.log and zxcom.log files in the
/home/zxin10/log directory and check the contents of these
log files.
Standard for The zxcom.log and sdf.log files do not contain error records such
Reference as "error" and "failed".
Exception 1. If the intelligent network is restarted, check whether it is
Handling restarted for a good reason, for example, upgrade. If it is
restarted due to faults, contact ZTE technical support engi-
neers.

2. If the persistent connection is broken, check the opposite pro-

gram or the network.
3. If the connection to the database is broken or re-established,
check the database or network.
4. If there are timer errors, check the timer.
5. If zxmoni determines exceptions on application processes,
check the corresponding application programs.
6. Send the downloaded log files to ZTE technical support engi-
neers.
4.2.3 Backing Up Application

Programs and Log Files
Purpose Back up OMMP application programs and log files for recovery and
troubleshooting in time of failure.
mand:
$tar -cvf /home/zxin10.tar *
Transfer the zxin10.tar file in binary mode through FTP to the
backup server or your local disk.
Standard for A zxin10.tar backup file is generated in the /home/zxin10 direc-
Reference tory. This file is transferred to another host.
4.3 Parlay Server Checks

4.3.1 Parlay Server Maintenance Task
List

routine checks listed below on the Parlay Server.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
Application log of
5 4.3.2
Parlay Server

Daily Maintenance
No. Operation Guide
Item
7 4.3.3
4.3.2 Checking Parlay Server Logs
Purpose Discover potential problems of the Parlay Server by checking Par-

lay Server logs.
Instruction Log in to the Parlay Server as user "zxin10". Go to the
/home/zxin10/log directory. Check the zxcom.log and sdf.log
files.
Standard for The zxcom.log and sdf.log files do not contain error records such
Reference as "error" and "failed".
Exception 1. If the intelligent network is restarted, check whether it is
Handling restarted for a good reason, for example, upgrade. If it is
restarted due to faults, contact ZTE technical support engi-
neers.
2. If the persistent connection is broken, check the opposite pro-
gram or the network.
3. If the connection to the database is broken or re-established,
check the database or network.
4. If there are timer errors, check the timer.
5. If zxmoni determines exceptions on application processes,
check the corresponding application programs.

Purpose Back up Parlay Server application programs and log files for re-
covery and troubleshooting in time of failure.
command:
$tar -cvf /home/zxin10.tar *
Transfer the zxin10.tar file in binary mode through FTP to the
backup server or your local disk.
Standard for A zxin10.tar backup file is generated in the /home/zxin10 direc-
Reference tory. This file is transferred to another host.

4.4 SIP Proxy Server Checks

4.4.1 SIP Proxy Server Maintenance
Task List

routine checks listed below on the SIP Proxy server.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
Application log of SIP

5 4.4.2
Proxy
7 4.4.3
4.4.2 Checking SIP Proxy Logs
Purpose Make sure that SIP proxy log files do not contain error records.
Instruction 1. Log in to the SIP proxy server as user "zxin10" by using the
FTP tool.
2. Download the sipproxy.log and zxcom.log files in the
/home/zxin10/log directory and check the contents of these
log files.
Standard for The sipproxy.log and zxcom.log files do not contain error
Reference records such as "error" and "failed".
Handling

Purpose Restore application programs when the application programs fail.

Instruction 1. Log in to the SIP proxy server as user "root".
2. Run the following commands:
#cd /home

#tar -cvf zxin10.tar zxin10/*

Standard for The zxin10.tar file is present in the /home directory.
Reference
4.5 EBASE Server Checks

4.5.1 EBASE Server Maintenance
Task List

routine checks listed below on the EBASE server.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
Application log of
5 4.5.2
EBASE
4.5.2 Checking EBASE Logs
Purpose Make sure that EBASE logs do not contain error records.
Instruction 1. Log in to the EBASE server as user "zxin10" by using the FTP
tool.
2. Download the zxcom.log file in the /home/zxin10/log direc-
tory and check the contents of the log file.
Standard for The zxcom.log file does not contain error records such as "error"
Reference and "failed".

Exception Send the downloaded log file to ZTE technical support engineers.
Handling
4.6 Smart Call Alert Checks

4.6.1 Smart Call Alert Maintenance
Task List

routine checks listed below on the Smart Call Alert.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
Application log of
5 4.6.2
Smart Call Alert
7 4.6.3
4.6.2 Checking Smart Call Alert Logs
Purpose Discover potential problems of the Smart Call Alert by checking

Smart Call Alert logs.
Instruction 1. Log in to the Smart Call Alert server as user "zxin10" by using
the FTP tool.
2. Download the smtrans.log file in the /home/zxin10/sm-
trans/trace directory and check the contents of the log file.
Standard for The smtrans.logfile does not contain error records such as "error"

Exception Send the downloaded log file to ZTE technical support engineers.
Handling

Programs and Configuration
Files
Purpose Back up application programs and configuration files for recovery.
Instruction To back up the /smtrans installation directory, perform the follow-
ing steps:
1. Telnet to the Smart Call Alert server and go to the
/home/zxin10 directory.
2. Run the $tar -cvf smtransbak.tar ./smtrans/* command and transfer
the smtransbak.tar file to the backup server in binary mode
through FTP.
Standard for The smtransbak.tar file is present on the backup server.
Reference
Exception If the backup is not successful, perform another backup or contact
Handling ZTE technical support engineers.
4.7 VGW Checks

4.7.1 VGW Maintenance Task List

routine checks listed below on the VGW.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
5 Application log of VGW 4.7.2
7 4.7.3
Monitoring program of
8 4.7.4
VGW process

4.7.2 Checking VGW Logs
Purpose Make sure that the VGW program runs normally.

Instruction 1. Log in to the VGW server as user "zxin10" by using the FTP
tool.
2. Download the smsagent.log file in the /home/zxin10/sm-
sagent/trace directory and check the contents of the log file.
Criteria Log files do not contain error records such as "error" and "failed".
Handling

Programs and Configuration
Files
Purpose Back up application programs and configuration files for recovery.
Instruction On the SUSE Linux operating system, telnet to the VGW server and
go to the /home/zxin10 directory. Run the $tar -cvf smsagentbak.tar
./smsagent/* command to compress the /home/zxin10/smsagent di-
rectory. Transfer the smsagentbak.tar file to the backup server
in binary mode through FTP.
Criteria The smsagentbak.tar file is present on the backup server.
4.7.4 Checking the Monitoring

Program
Purpose Check whether the watchdog is running normally.

Instruction Log in as user "zxin10" and run the $ps -ef | grep s10dog com-
mand. The following information is displayed:
root 28630 28397 0 Apr01 ? 00:02:23 ./s10dog
Criteria The s10dog process exists.

Exception Run the ./s10dog command in the installation directory, for ex-
Handling ample, /home/zxin10/smsagent/bin.


Checks
routine checks listed below on the Oracle database server.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
5 Database log 4.1.6
4.9 Web Server Checks

4.9.1 Web Server Maintenance Task
List

routine checks listed below on the Web server.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
5 Application log of Web 4.9.2
7 4.9.3
4.9.2 Checking Web Service Logs
Purpose Make sure that the Web service runs normally.

Instruction ¡ To check Tomcat logs, perform the following steps:

i. 1. Log in to the Web server as user "root".
ii. 2. Go to the /home/zxin10/was/tomcat/logs directory.
#cd /home/zxin10/was/tomcat/logs
iii. 3. Check the catalina.out file.
#tail -f catalina.out
¡ To check JBoss logs, perform the following steps:
ii. 2. Go to the /usr/java/jboss/server/default/log di-
rectory.
#cd /usr/java/jboss/server/default/log
#tail -f server.log
Standard for ¡ Normal Tomcat logs are as follows:
Reference
soap check Language=""
soap check Language: Language="" or Language is no existed
according to the name of Language
soap check Language: will user default language!
MWNDest is empty, use default:2197850127
EntryAuthen is empty, use default:1
MessageNotify is empty, use default:1000
UserName is empty, use default:
Email is empty, use default:
ZipCode is empty, use default:
IDNumber is empty, use default:
soap servcie:CreateUser begin:
soap servcie:DeleteUser check parameter:
remote ip: 10.26.73.105
IP check is off.
UserNumber:7199086180
OperID:vivovms
OperPwd:vivovms
soap servcie:CreateUser:write log
soap servcie:DeleteUser begin:
soap servcie:CreateUser:2197850127 end
remote ip: 10.26.73.105
IP check is off.
OperID:vivovms
OperPwd:vivovms
soap servcie:DeleteUser begin:
remote ip: 10.26.73.105
IP check is off.

OperID:vivovms
OperPwd:vivovms
¡ Normal JBoss logs are as follows:
2009-06-11 02:39:24,290 INFO [com.zte.ngmail.serv-

ice.mailmg.MailManageBean] receive response:4591142838
ice.mailmg.MailManageBean] receive response code:0
2009-06-11 02:39:24,362 DEBUG [com.zte.ngmail.serv-
ice.mailmg.MailManageBean] create mail box 1997358239 over.
return value: 0
2009-06-11 02:39:24,380 DEBUG [com.zte.ngmail.serv-
ice.mailmg.MailManageBean] delete mail box 1198644014 over.
return value: 0
2009-06-11 02:39:24,380 WARN [com.zte.ngmail.serv-
ice.mailmg.MailManageBean] &&&&&&&& have sent
UnregUserMessage
ice.mailmg.MailManageBean] send message event:24503
ice.mailmg.MailManageBean] send message body:1198644014
Exception If there are exceptions, switch the Web cluster and contact ZTE

Purpose Restore application programs after they are damaged or deleted.

Restore log files after they are damaged or deleted.
Instruction ¡ To back up Tomcat application programs and logs, perform the
following steps:
i. Log in to the Web server as user "root".
#cd /home/zxin10
#tar -cvf was_bak.tar was/*
¡ To back up JBoss application programs and logs, perform the
following steps:
#cd /usr/java
#tar -cvf jboss_bak.tar jboss/*
Standard for ¡ To check the backup files for Tomcat application programs and
Reference logs, perform the following steps:


#cd /home/zxin10
#ls
Check whether the was_bak.tar file is present.
¡ To check the backup files for JBoss application programs and
logs, perform the following steps:
ii. 2. Run the following commands:
#cd /usr/java
#ls
Check whether the jboss_bak.tar file is present.
4.10 UMA Checks

4.10.1UMA Maintenance Task List

routine checks listed below on the UMA.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
5 Application log of UMA 4.10.2
Monitoring program of
7 4.10.3
UMA process
4.10.2Checking UMA Logs
Purpose Check whether the UMA service runs normally.

Instruction 1. Log in to the UMA server as user "zxuma" by using the FTP
tool.
2. Download and check UMA log files. All UMA log files are de-
scribed in the table below.

Log File Path Description
Inter-module
zxcom.log /home/zxuma/log
communication logs
* (stored on a daily /home/zxuma/log/s-

UMA system log
basis) ys/
Message log
Message logs are
normally written
to the database
* (stored by protocol /home/zxuma/log- directly. They are
and message type) /data/ written to the data
directory only when
the rept.datalog
parameter is set to
1.
Standard for ¡ The zxcom.log file does not contain error records such as "er-
Reference ror" and "failed" and does not show a process is restarted re-
peatedly.
¡ The latest log file in the /home/zxuma/log/sys directory does
not contain error records such as "error" and "failed".
¡ Message log files are generated in the /home/zxuma/log/data
directory.
Handling
4.10.3Checking the Monitoring

Program
Purpose Check whether the monitoring process runs well.

Instruction Check the status of the monitoring process zxmoni by using the
following commands:
#su - zxuma
$ps -ef | grep zxmoni
Standard for The zxmoni process runs normally, as shown below.
Reference
$ps -ef | grep zxmoni
zxuma 17859 1 0 10:48 pts/6 00:00:00 zxmoni
The PID of the zxmoni process is 17859 and unchanged.
Exception If the zxmoni process is not started, start the zxmoni process
Handling with the assistance of ZTE technical support engineers when the
service traffic is not high.


4.11.1Mail Processor Maintenance
Task List

routine checks listed below on the Mail Processor.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
Application log of Mail

5 4.11.2
Processor
7 4.11.3
8 Mail Processor and 4.11.4
other servers
4.11.2Checking Mail Processor Logs
Purpose Make sure that the Mail Processor programs run normally and logs
do not contain errors.
Instruction 1. Log in to the Mail Processor as user "root" through FTP.
2. Download the zxcom.log file in the /usr/zte/mdss/log direc-
tory and check the contents of the log file.
Standard for The zxcom.log file does not contain error records such as "error"
Exception Send the zxcom.log file to ZTE technical support engineers.
Handling
4.11.3Backing Up Application
Purpose Restore application programs when the application programs fail.


2. Run the following commands:

#cd /usr/zte
#tar -cvf mdss.tar ./mdss/*
Standard for The mdss.tar file is present in the /usr/zte directory.
Reference
4.11.4Checking Connections Between

the Mail Processor and Other
Servers
Purpose Make sure that the connection between the MDSS server and the
Parlay Server is normal.
#netstat -an|grep <Mail Processor IP address>
Standard for The connection between the Mail Processor and port 5000 of the
Reference Parlay Server is established, as shown below.
Exception 1. Check whether the /usr/zte/MDSS/etc/win_mgt.ini file is

Handling correct.
2. Check whether the network is normal by pinging the IP address
of the Parlay Server.
4.12 Mail Clear Server Checks

routine checks listed below on the Mail Clear server.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
5 Application log of IMP 4.1.7

4.13 Sybase Database Server

Checks
routine checks listed below on the Sybase database server.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
5 Database log 4.1.6

routine checks listed below on the statistics server.
Daily Maintenance
No. Operation Guide
Item
2 OS patch 4.1.2
4 OS log 4.1.4
5 Application log of IMP 4.1.7
4.15 Firewall Checks

4.15.1Firewall Maintenance Task List

routine checks listed below on the firewall.

Daily Maintenance
No. Operation Guide
Item
1 Firewall log 4.15.2
2 Firewall alarm 4.15.3
4.15.2Checking Firewall Logs
Purpose Monitor the firewall.

Instruction ¡ Juniper
i. 1. Log in to the firewall through IE as user "netscreen".
ii. 2. Select Reports > System Log > Event. Select a sever-
ity level from the Log Level pull-down list box.
iii. 3. Check the logs of the following levels: emergency, alert,
critical, error and warning.
¡ Pix
i. Telnet to any server and show firewall logs by running the
following command:
#get log event
The output is as shown below.
ii. If SUSE output is enabled, log in to the SUSE syslog server

as user "root" and run the following system:
#more /var/log/pix.log

Standard for ¡ Juniper

Reference
The logs do not contain the following levels: emergency, alert,
critical, error and warning. Or the logs of those levels have
been checked.
¡ Pix
There are no error logs.
If SUSE output is enabled, the pix.log file does not contain
unchecked error logs.
Exception ¡ Juniper
Handling
To download event logs, perform the following steps:
In this example, download event logs to the
C:\netscreen\logs path. The file name is evnt07-02.txt.
i. Select Reports > System Log > Event. Click Save.
ii. Select the Save option, and then click OK.
iii. Set the path to C:\netscreen\logs, the file name to
evnt07-02.txt, and then click Save.
iv. Send the evnt07-02.txt file to ZTE technical support en-
gineers.
¡ Pix
Handle the fault based on logs. If the fault persists, save logs
and contact ZTE technical support engineers.
4.15.3Checking Firewall Alarms
Purpose Monitor the firewall.

Instruction Telnet to any server and show firewall alarms by running the fol-
lowing command:
#get alarm event

Standard for There are no alarms.

Reference
Exception Handle the fault based on alarms. If the fault persists, save alarms
Handling and contact ZTE technical support engineers.
4.16 Layer 3 Switch and GRE

Router Checks
4.16.1Layer 3 Switch and GRE Router
Maintenance Task List

routine checks listed below on the layer 3 switch and GRE router.
Daily Maintenance
No. Operation Guide
Item
Network interface
1 4.16.2
status
2 VLAN 4.16.3
3 Route table 4.16.4
4 F5 status 4.16.5
4.16.2Checking Interfaces
Purpose Check the status of the interfaces of the switch.

Instruction Check interfaces by running the following command:
#show interfaces

Standard for The status of the interfaces is "Up".

Reference
Exception Check the interfaces. Check whether the optical fibers and net-
Handling work cables are normal. Check whether the opposite devices are
powered on.
4.16.3Checking VLANs
Purpose Check the status of the VLANs.

Instruction Check VLANs by running the following command:
#show vlan

Standard for There are VLANs. Interfaces are configured in each VLAN.
Reference
Handling
4.16.4Checking the Routing Table
Purpose Check the routes configured on the switch.

Instruction Run the following command to check the routing table:
#show ip route

Standard for The normal output is as shown in the figure above.

Reference
Exception 1. Check whether routes and IP addresses are configured cor-
Handling rectly.
2. Delete VRRP cache.
4.16.5Checking Big-IP Logs of the F5

Device
Purpose Check switch logs.

Instruction 1. Enter the Web address of the F5 device in the IE address bar
and go to the interface of the F5 device.
2. Select LOG FILES in the navigation bar.
3. Click the BIG-IP Log tab

Standard for The logs can show switch-over operations.

Reference
If there are no logs, this indicates that the F5 device works nor-
mally during the latest seven days, and all the nodes work nor-
mally. No operations are performed and all the logs are deleted.
Exception Handle the fault based on logs. Send the logs you cannot under-
Handling stand to ZTE technical support engineers.


�5 Quarterly Maintenance
>> Checking Power Connections
>> Cleaning Cabinets
>> Removing Foreign Objects
>> Checking Ground Wires
>> Testing Grounding Resistance
>> Checking Fans
>> Cleaning Tape Drives
5.1 Checking Power Connections

Instruction Check whether cables are connected to sockets well.
Check the power distribution cabinet to see if the output voltage
is within the normal range.
Standard for Cables are connected to sockets well. Air switches in appropriate
Reference capacities are installed for each power supply. Cables are routed
to beneath the floor in the position of the cabinet. Power connector
boards agree with the cabinet and 10~16 A is required.
AC voltage: 220 V +4%, -8% (198~232 V)
AC frequency: 50Hz +/- 0.5Hz
DC voltage: -18V +5%, -5%
Exception If the output voltage of the power distribution cabinet is beyond
Handling the normal range, contact ZTE technical support engineers. Adjust
the voltage to the normal range or replace power devices.
5.2 Cleaning Cabinets

Instruction Check if there is any dirt on the surface of the cabinet.
Check if there is dust on the dust-screen at the bottom of the
cabinet.
Check if there is rust in the cabinet.
Standard for There is no dirt on the surface of the cabinet.
Reference
There is no dust on the dust-screen at the bottom of the cabinet.
There is no rust in the cabinet.

Exception If any dirt, dust, or rust is found, remove it.

Handling
5.3 Removing Foreign Objects

Instruction Check the inside of racks to see if there is any foreign objects in
the racks.
Standard for No foreign object is found.
Reference
Exception Take out foreign objects if any.
Handling
5.4 Checking Ground Wires

Instruction Check whether the ground wire is connected securely and whether
there is any rust.
Standard for The ground wire is connected securely without any rust.
Reference
Exception If the ground wire is not connected securely or there is rust, secure
Handling the connection or remove the rust.
5.5 Testing Grounding

Resistance
Instruction Test the grounding resistance in the equipment room regularly to
see if it meets the requirements.
Standard for Ground wires are insulated and completely separated from any
Reference conductor wire.
The diameter of ground wires is at least 3.5mm.
Ground wires are not used as neutral wires and separated from
neutral wires.
The grounding resistance measured at the socket between the
neutral wire and the live wire is less than 2 Ω.
The voltage measured at the socket between the neutral wire and
the ground wire is less than 1V.
Iron pipes cannot be used as ground wires.
The grounding resistance is less than 1 Ω at the grounding end of
ground wires.
Exception If the grounding resistance is beyond the required range, take
Handling measures to lower the grounding resistance.

Chapter 5 Quarterly Maintenance
5.6 Checking Fans

Instruction Check if each fan is running and if the sound is normal.
Standard for All the fans are running normally and there is no abnormal sound,
Reference for example, blade scratching the tray.
Exception If you find any fan does not run, or the sound is abnormal, contact
Handling ZTE technical support engineer immediately to locate the fault and
recover the fan system.
5.7 Cleaning Tape Drives

Instruction Insert a cleaning tape into the tape drive, clean the drive for
around 30 seconds and then record the operation.
Standard for Tape drives are cleaned successfully.
Reference
Exception If any exception is detected, contact ZTE technical support engi-
Handling neers.

NGCSS SM09 E1 0 NGCSS Routine Maintenance

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

NGCSS SM09 E1 0 NGCSS Routine Maintenance

Uploaded by

Copyright:

Available Formats

NGCSS

Copyright © 2006 ZTE CORPORATION.

The ultimate right to interpret this product resides in ZTE CORPORATION.

Publishing Date (MONTH/DATE/YEAR) : 11/10/2010

will be able to:

Confidential and Proprietary Information of ZTE CORPORATION 1

This page is intentionally blank.

2 Confidential and Proprietary Information of ZTE CORPORATION

1.1 Introduction to Routine

Confidential and Proprietary Information of ZTE CORPORATION 3

1.2 Requirements for the

4 Confidential and Proprietary Information of ZTE CORPORATION

Product ¡ Hardware structure and performance parameters of the NGCSS

1.3 Precautions in Routine

Confidential and Proprietary Information of ZTE CORPORATION 5

and marked with a sign. Commonly used parts must be supplied

1.4 Common Maintenance

6 Confidential and Proprietary Information of ZTE CORPORATION

1.5 Maintenance Tools

Confidential and Proprietary Information of ZTE CORPORATION 7

This page is intentionally blank.

8 Confidential and Proprietary Information of ZTE CORPORATION

2.1 Universal Daily Maintenance

Purpose Check the equipment room to make sure it meets requirements.

Confidential and Proprietary Information of ZTE CORPORATION 9

¡ Check if devices in the equipment room are properly grounded.

2.1.2 Checking Hardware Indicators

10 Confidential and Proprietary Information of ZTE CORPORATION

2.1.3 Checking Disks

Confidential and Proprietary Information of ZTE CORPORATION 11

the Computer Management Window, select Storage

2.1.4 Checking Database Space

12 Confidential and Proprietary Information of ZTE CORPORATION

create table tmp(name varchar(50),rows int,reserved varchar(50),data

2.2 OMMP Checks

During daily maintenance on the NGCSS, you need to perform rou-

2 SMAP alarm 2.2.2

Maximum daily call

2.2.2 Checking SMAP Alarms

Purpose Discover the potential problems of the intelligent network by

2.2.3 Checking Daily Peak Traffic

Confidential and Proprietary Information of ZTE CORPORATION 13

2.3 SIP Proxy Server Check

2.4 Oracle Database Server

2 Disk status 2.1.3

3 Database space 2.1.4

2.5 Parlay Server Check

14 Confidential and Proprietary Information of ZTE CORPORATION

2.6 EBASE Server Check

2.7 Notify Scheduler Check

2.8 UMA Check

2.9 Mail Processor Check

2.10 Mail Clear Server Check

2.11 Smart Call Alert Check

2.12 VGW Check

Confidential and Proprietary Information of ZTE CORPORATION 15

2.13 Media Server (CMS9000/3000)

During daily maintenance on the NGCSS, you need to perform rou-

2 Media server alarm 2.13.3

2.13.2Checking Media Server

Purpose Make sure that no hardware alarms occur on the CMS9000/3000.

2.13.3Checking Media Server Alarms

Purpose Make sure that the CMS9000/3000 works normally.

16 Confidential and Proprietary Information of ZTE CORPORATION