Troubleshooting 10 (Linux)

AAA
V200R002C11
Troubleshooting - Linux
Issue 10
Date 2014-03-14
HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2014. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior written
consent of Huawei Technologies Co., Ltd.
Trademarks and Permissions
and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective holders.
Notice
The purchased products, services and features are stipulated by the contract made between Huawei and the
customer. All or part of the products, services and features described in this document may not be within the
purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information,
and recommendations in this document are provided "AS IS" without warranties, guarantees or representations
of any kind, either express or implied.
The information in this document is subject to change without notice. Every effort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, express or implied.
Huawei Technologies Co., Ltd.

Address: Huawei Industrial Base
Bantian, Longgang
Shenzhen 518129
People's Republic of China
Website: http://www.huawei.com
Email: support@huawei.com
Issue 10 (2014-03-14) Huawei Proprietary and Confidential i

Copyright © Huawei Technologies Co., Ltd.
AAA
Troubleshooting - Linux About This Document
About This Document
Purpose
This document describes the faults that may occur on the modules of the AAA system, and how
to remove the faults.
This document describes the classification of troubleshooting, flow for troubleshooting and fault
analysis methods. It also provides typical cases of the faults on each module of the AAA.
Intended Audience
The intended audiences of this document are:
System Maintenance Engineer
Symbol Conventions
The symbols that may be found in this document are defined as follows.
Symbol Description
Indicates an imminently hazardous situation which, if not

avoided, will result in death or serious injury.
DANGER
Indicates a potentially hazardous situation which, if not

avoided, could result in death or serious injury.
WARNING

avoided, may result in minor or moderate injury.
Issue 10 (2014-03-14) Huawei Proprietary and Confidential ii

AAA
Symbol Description

avoided, could result in equipment damage, data loss,
performance deterioration, or unanticipated results.
NOTICE is used to address practices not related to
personal injury.
NOTE Calls attention to important information, best practices and

tips.
NOTE is used to address information not related to
personal injury, equipment damage, and environment
deterioration.
Privacy Notice
To find the fault reasons, necessary service logs may need to be collected, and the logs may
contain subscriber information such as the subscriber number. You must use the information
according to laws and local carrier requirements, and are not allowed to send the log information
out of the customer's network. During use, you are advised to use the anonymization tool to hide
subscriber information. When related faults are located, you must delete the logs in a timely
manner.
Update History
Updates between document versions are cumulative. Therefore, the latest document version
contains all updates made to previous versions.
Updates in Issue 10 (2014-03-14)

The tenth commercial release has the following updates:
6.2 Solution is updated.

The ninth commercial release has the following updates:
B.1 setval Commands is updated.

The eighth commercial release has the following updates:
Issue 10 (2014-03-14) Huawei Proprietary and Confidential iii

AAA
The product version V200R002 is changed to R002C10.

The seventh commercial release has the following updates:
The product name infoX AAA is changed to HUAWEI AAA.

The sixth commercial release has the following updates:
The product name infoX-AAA is changed to infoX AAA.

The fifth commercial release has the following updates:
Modify the test questions.

The fourth commercial release has the following updates:
Modify the test questions.

The third commercial release has the following updates:
Information about the Linux platform and DB2 database application environment is added.

The second commercial release has the following updates:
"Appendix C The setval Command Reference" is added.

Initial commercial release.
Issue 10 (2014-03-14) Huawei Proprietary and Confidential iv

AAA
Troubleshooting - Linux Contents
Contents
About This Document.....................................................................................................................ii

1 Overview.........................................................................................................................................1
1.1 Classification of Troubleshooting .................................................................................................................................2
1.2 Flow for Troubleshooting ..............................................................................................................................................2
1.2.1 Flow for Troubleshooting ...........................................................................................................................................2
1.2.2 Collecting Information ...............................................................................................................................................3
1.2.3 Analyzing Faults .........................................................................................................................................................4
1.2.4 Rectifying Faults .........................................................................................................................................................4
1.3 Fault Analysis Methods..................................................................................................................................................4
1.3.1 Analyzing the Original Information ...........................................................................................................................4
1.3.2 Analyzing Indicator Status .........................................................................................................................................4
1.3.3 Analyzing the Alarm Information ..............................................................................................................................4
1.3.4 Analyzing the Log Information ..................................................................................................................................5
1.3.5 Tracing and Analyzing Messages ...............................................................................................................................5
2 Troubleshooting the iSCC1.........................................................................................................6

2.1 Introduction ...................................................................................................................................................................7
2.2 Solution ..........................................................................................................................................................................7
2.3 Typical Cases .................................................................................................................................................................7
3 Troubleshooting the iSCC2.........................................................................................................8

3.1 Introduction ...................................................................................................................................................................9
3.2 Solution ..........................................................................................................................................................................9
3.3 Typical Cases................................................................................................................................................................11
3.3.1 Authenticating the Subscriber Fails Because of Overloaded iSCC2 ........................................................................11
4 Troubleshooting the MML Server...........................................................................................13

4.1 Introduction .................................................................................................................................................................14
4.2 Solution ........................................................................................................................................................................14
4.3 Typical Cases................................................................................................................................................................15
4.3.1 Billing System Fails to Define a Subscriber .............................................................................................................15
4.3.2 Unlocking the Operator root .....................................................................................................................................15
5 Troubleshooting the WMAS.....................................................................................................17
Issue 10 (2014-03-14) Huawei Proprietary and Confidential v

AAA
Troubleshooting - Linux Contents
5.1 Introduction .................................................................................................................................................................18

5.2 Solution ........................................................................................................................................................................18
5.3 Typical Cases................................................................................................................................................................18
5.3.1 Defining the Subscriber Fails ...................................................................................................................................18
5.3.2 Logging In to the WMAS Fails ................................................................................................................................19
6 Troubleshooting the Billproc....................................................................................................20

6.1 Introduction .................................................................................................................................................................21
6.2 Solution.........................................................................................................................................................................21
6.3 Typical Cases................................................................................................................................................................22
6.3.1 Loading the Bill into Database Fails Because of Insufficient Data Space ...............................................................22
7 Troubleshooting the Portal........................................................................................................24

7.1 Overview......................................................................................................................................................................25
7.2 Typical Cases................................................................................................................................................................25
7.2.1 Failed to Obtain an IP Address Due to Incorrect BRAS Configuration....................................................................25
7.2.2 Failed to Display the Portal Login Page Due to Overlarge Log File or Connection Number...................................26
A Troubleshooting the Network.................................................................................................28

A.1 Overview......................................................................................................................................................................29
A.2 Solution........................................................................................................................................................................29
A.3 Typical Cases...............................................................................................................................................................29
A.3.1 Firewall not Supporting Session Update..................................................................................................................29
A.3.2 Description of Ports Mapping on Firewall...............................................................................................................29
A.3.3 Subscriber Fails to Go Online After Switchover of the Two-Node Cluster.............................................................30
A.3.4 Subscriber Fails to Go Online Caused by Device Address Change.........................................................................31
A.3.5 Subscriber Fails to Go Online Caused by Different Shared Keys Between Device and the AAA..........................32
B The setval Command Reference..............................................................................................33

B.1 setval Commands.........................................................................................................................................................34
B.2 Module List..................................................................................................................................................................35
C Acronyms and Abbreviations..................................................................................................37

Index..................................................................................................................................................39
Issue 10 (2014-03-14) Huawei Proprietary and Confidential vi

AAA
Troubleshooting - Linux 1 Overview
1 Overview
About This Chapter
1.1 Classification of Troubleshooting
1.2 Flow for Troubleshooting
1.3 Fault Analysis Methods
Issue 10 (2014-03-14) Huawei Proprietary and Confidential 1

AAA
1.1 Classification of Troubleshooting

Troubleshooting is a type of remedial maintenance. Its purpose is to rectify faults quickly in the
case of failure, recover the system, and try to minimize the loss caused by faults.
Based on the emergency degree and influence scope, the troubleshooting is classified into two
types:
l General troubleshooting
General troubleshooting is used to remove faults that are not emergent and only have a
slight impact in a small scope, or those that need not be handled at once due to good backup
measures, such as two-node switchover.
l Emergency troubleshooting
Emergency troubleshooting is used to remove the emergent fault that may have a severe
impact, such as system down.
1.2 Flow for Troubleshooting
1.2.1 Flow for Troubleshooting

If emergent faults occur in the AAA, follow the steps as listed inTable 1-1.
Table 1-1 Troubleshooting flow
Number Task Action
1 Back up the database Prepare before the troubleshooting.

completely.
2 Back up the logs such as the Prepare before the troubleshooting.

debugging log, the core file,
the operation system log, and
the DBMS log.
3 Check the application system. If the faults are found, install the application
system again or contact Huawei
Technologies Co., Ltd.
4 Check the two-node cluster. If the faults are found, switch to the standby
node and restart, or contact Huawei
5 Check the nodes of the cluster. If the faults are found, contact Huawei

AAA
Number Task Action
6 Check the nodes of the two- If the faults are found on a node, remove the
node cluster. node from the Load Balance Device (LBD).
After the troubleshooting, reload the node
to the LBD.
7 Check whether the disk space If the disk space is full, release the space.
is full.
8 Check whether the network If the faults are found, contact Huawei
route is normal. technical support engineers.
9 Check whether the hardware If the faults are found, contact Huawei
devices are normal. technical support engineers.
10 Check whether the running of If the faults are found, refer to the AAA
the devices (including the Emergency Maintenance and contact
database and operation Huawei Technologies Co., Ltd.
system) is normal.
1.2.2 Collecting Information
Before you handle a fault, try to collect the original information related to this fault as much as
possible.
Information Collection Approaches

You must collect the original fault information as much as possible. There are mainly three
sources for collecting the fault information:
l Faults reported by users or the customer service center

l Faults reported by other maintenance personnel
l Alarms output by the alarm system
Information Collection Methods

You should learn some skills for collecting the information.
The skills are as follows:
l Make sure that you are ready to collect data when required. If any fault, a severe fault in
particular, occurs, understand the situation before proceeding any further. Do not handle
the fault blindly.
l Study and master the services, familiarize yourself with the system principles and service
flows in particular so that the root causes can be found at the earliest.
l When answering a fault claim call, the maintainance personnel should guide the user with
flexible methods, try to ask the subscriber from multiple aspects and angles, and get more
information about the fault.

AAA
l Strengthen the business contact inside and outside the department, and set up a good
business relationship with the maintenance personnel of other offices.
1.2.3 Analyzing Faults

The course of analyzing a fault is to find out the real cause from many possible causes. You can
determine the fault coverage, fault type, cause, and handling method to analyze a fault. The
common methods for fault analysis are described in section1.3 Fault Analysis Methods.
Fault analysis is the most important and the most difficult step in the troubleshooting flow. You
should have sound technical knowledge and rich experience. You should also improve the fault
handling skills in practice.
1.2.4 Rectifying Faults

The course of removing a fault is to take proper measures or follow correct steps to remove the
fault and recover the system. For example, to check the circuit, to replace the hardware, to change
the configuration data, and to restart the system.
For more information about fault removal, refer to the regulations of the customer, or the
operation guide provided by Huawei. For example, you can read the related documentation such
as the hardware replacement precautions and the system restart precautions.
1.3 Fault Analysis Methods
1.3.1 Analyzing the Original Information

Original information refers to the fault information reported by subscribers or displayed in
system alarms. It also refers to the onsite information collected in the earlier phase by
maintenance personnel through various channels. The original information is important for the
fault analysis.
The purpose of analyzing the original information is to judge the fault coverage, and determine
the fault type. This is the basis for locating the fault.
1.3.2 Analyzing Indicator Status

The hardware system (such as the two-node cluster and storage system) provides indicators that
indicate the running status and feature of the hardware. The indicators can directly reflect the
running status of the hardware. Thus, the indicator status is an important basis for fault analysis.
This method can be used to find out the causes for hardware system faults quickly. As the
information given by the indicators is not enough, it is recommended that this method be used
together with the method of analyzing the alarm information.
1.3.3 Analyzing the Alarm Information

The alarm information is output by the AAA, and is usually provided to the maintenance
personnel in the forms of indicator light, sound and screen output.

AAA
The alarm information contains fault symptoms, possible causes and handling suggestions. It is
also an important basis for fault analysis. Through this information, you can locate faults and
find out possible causes.
1.3.4 Analyzing the Log Information

Through the logs provided by the AAA, you can learn the system running status. Compared with
the alarm information, log information is more specific. It can be directly used to locate faults
and find out the causes.
The AAA defines logs at five levels: prompt, alarm, minor error, major error, and fatal error. It
also provides the level-based filter function. You can set the log output levels as required to get
more information for fault location.
1.3.5 Tracing and Analyzing Messages

This method can be used to analyze the login flow, and the inter-device message interaction
flow. By using the traced results as clues for further analysis, you can directly locate faults and
find out possible causes.

AAA
Troubleshooting - Linux 2 Troubleshooting the iSCC1
2 Troubleshooting the iSCC1
About This Chapter
2.1 Introduction
2.2 Solution
2.3 Typical Cases

AAA
2.1 Introduction
The main functions of the iSCC1 module is to deduct the basic fee, record the bill of the basic
fee, update the charge policy, scan and record the abnormal user regularly.
2.2 Solution
Check the Process

The iSCC1 module contains only an independent process. The system timer calls the process to
fulfill the related tasks regularly.
Log in to the system as user iscc1. Run the p command to check whether the process is normal.
In normal status, the following Java process exists:

iscc1 22234 1 0 02:43:36 ? 0:10 /usr/jdk1.4/bin/java -server -Dacc
-Dsun.rmi.dgc.server.gcI nterval=0x7FFFFFFFFF
Check the Log

Independent process log file: ~iscc1/log/AccIndepProcess/AccIndepProcess*.log
Check the Configuration File

Independent process configuration file: ~iscc1/config/accindepprocess.cfg
2.3 Typical Cases

None.

AAA
3 Troubleshooting the iSCC2
About This Chapter
3.1 Introduction
3.2 Solution
3.3 Typical Cases

AAA
3.1 Introduction
The function of the iSCC2 is to control services. The iSMS system manages the iSCC2.
The iSCC2 system fault includes the following:
l Authentication failure
l Authorization failure
l Accounting failure
3.2 Solution
Check the Process

Faults in the iSCC2 process may cause failures such as authentication failure, subscriber going
offline, charging error, and failure to go offline.
Log in to the system as user iscc2. Run the p command to check the iSCC2 process. The iSCC2
module processes include manager, scfserver, scf (may be more than one), and sdf.
The iSCC2 process has faults if:
l The number of the iSCC2 processes is incorrect.

l The memory usage is excessive (over 100 MB).
l The CPU usage is excessive (over 15%).
Check the Log

The ~iscc2/log directory stores the following iSCC2 log files:
l The scf process log: U_scfn.log (n is the serial number of the scf process. The number is
an integer 0 or more than 0.)
l The manager process log: U_manager.log
l The scfserver process log: U_server.log
l The sdf process log: U_sdf.log
l The debug log: After the Debug switch is turned on, the system generates this log. This log
is used to record the debug log of the setval tool.
NOTICE
The debug log function affects system performance, slows down the service running speed,
and occupies disk space. Do not enable the function when the system is in commercial use.
If you want to enable this function to locate a fault, you must have your supervisor's
permission. After locating the fault, disable the function as soon as possible.

AAA
– U_scfn_debug.log: n is the serial number of the scf process. The number is an integer
equals to or more than 0.
– U_scfsrv_debug.log
– U_sdf_debug.log
– U_man_debug.log
Check the Configuration File

The iSCC2 configuration files are stored in the ~iscc2/config directory. The configuration files
include the following:
l SCPCFG.INI
It is used to set the number of accessed subscribers, authentication charging port, public
key, and performance parameters.
l SYSCFG.DAT
It is used to set the parameters of all the network devices connected to the iSCC2. The
network devices include the BRAS, PDSN, and GGSN.
l DiamBaseConfig.xml
It is used to set the parameter for connecting the AAA to the eGGSN that supports the Gy
interface.
Tracing
The AAA provides a message tracing tool setval to trace the subscriber authentication and
charging flow.
When you run the setval command, the system displays the Help information and command
syntax.
When you run the setval h or setval help command, the system displays the help information
and numbers of the level-1 module and level-2 module.
For details about the parameters, see B The setval Command Reference.
NOTICE
Use with caution the setval command that is used for tracing all the online users, because this
operation will affect the system performance and may lead to call barring.
Table 3-1lists the common setval commands.
Table 3-1 Common setval commands
Task Command
Print all the message setval u none;setval all;setval m 1;setval l 1;setval pt

logs. 100000000;setval ps 600

AAA
Task Command
Print the message log setval u user name;setval m 1;setval l 1;setval 1 255;setval pt
when a subscriber is 99999999;setval ps 600000;setval t 2
online.
Print the message setval l 2;setval m 1;setval 5 4;setval pt 999999999;setval ps

handling time. 9999999;setval t 2
Print the number of setval l 1;setval m 1;setval 5 6;setval pt 99999999;setval ps

machines processed in 600000;setval i 1;setval t 2
each scf process.
Print the detailed log setval l 1;setval m 1;setval 6 2;setval 6 3;setval pt

run by the SQL 999999999;setval ps 60;setval t 2
command.
Print the detailed logs setval l 2;setval m 1;setval 4 255;setval pt 999999999;setval ps

of the processes 9999999;setval t 2;setval l 2;setval m 1;setval 3 255;setval pt
scfserver, manager 999999999;setval ps 9999999;setval t 2;setval l 2;setval m
and sdf. 1;setval 2 255;setval pt 999999999;setval ps 9999999;setval t 2
The following section describes how to use the tool setval by considering the example of tracing
the online and offline flows of a subscriber.
1. Trace the specified subscribers' messages.

Run the following commands to trace the specified subscribers:
setval u user name;setval m 1;setval l 1;setval 1 255;setval pt 99999999;
setval ps 600000;setval t 2
2. Test whether the subscriber can go online and offline normally by using a subscriber
account.
3. Check the debugging log.
Access the ~iscc2/log directory and check the debugging message log (with debug in the
log file name). The name of the log file recording the online and offline processes of the
subscriber is U_scfn_debug.log (n is the serial number of the scf process).
3.3 Typical Cases
3.3.1 Authenticating the Subscriber Fails Because of Overloaded

iSCC2
Symptom
Some of the subscribers cannot pass the authentication. Subscribers that pass the authentication
can log in to the network.

AAA
Alarm
The "scf scsm buffer has been used off" alarm exists in the U_scfn_debug.log file of the iSCC2.
"System is busy, reject a ISNAuthRequest (STD)" is displayed in U_manager.log.

[iscc2] : /export/home/iscc2/log >more manager.log
(09:35:48) PROMPT Access number > 30
(09:35:48) PROMPT System is busy, reject a ISNAuthRequest(STD).
Cause
When the number of online subscribers exceeds a specified value, the iSCC triggers the load
control and denies new access to some subscribers to ensure the QoS.
Solution
Adjust the threshold value for the iSCC to enable the load control function by modifying the
following items in the ~iscc2/config/SCPCFG.INI file.
l maxDlgIDOfSCP
It is a maximum session permitted by the iSCC, that is, the maximum number of online
subscribers.
Value range: an integer equal to or greater than 0.
l MaxBufferOfTSCSM
It is the maximum number of the machines that can be saved in each scf process.
When the subscriber is accessed, the scf process creates and saves a machine. When the
subscriber goes offline, the scf cleans the mapping machine automatically.
Value range: 1 to the value of maxDlgIDOfSCP. The value cannot be smaller than the
quotient of maxDlgIDOfSCP divided by numOfSCDFRuning.
l numOfSCDFRuning
It is the number of the scf process to be started.
If many scf processes are started, the load of the access is shared by multiple processes. In
addition, the more scf processes that exist, the higher the demand of the machine
performance.
Value range: 1 to the value of maxDlgIDOfSCP.
The value cannot be smaller than the quotient of maxDlgIDOfSCP divided by
MaxBufferOfTSCSM.
l MaxAccess
It is the maximum number of the authentication requests accessible in a second. Requests
that exceed the value are discarded.
If the machine performs well, increase the value according to the actual situation.

AAA
Troubleshooting - Linux 4 Troubleshooting the MML Server
4 Troubleshooting the MML Server
About This Chapter
4.1 Introduction
4.2 Solution
4.3 Typical Cases

AAA
4.1 Introduction
The MML server is a service management system. It works with the WMAS to manage the
services in the Web mode.
The MML server fault is that the operator cannot normally perform operations normally on the
WMAS.
4.2 Solution
Check the Process
Log in to the system as user ismpmml. Run the procs.sh or p command to check whether the
process is normal.
Check the Port
Log in to the system as user ismpmml. Run the netstat –an|grep port number command to
check whether the status of the port 19999 and 20000 is ESTABLISHED.
Check the Log
To change the log level, do as follows:
Log in the system as user ismpmml. Run the release_off command to change the log level to
DEBUG.
NOTICE
The debug log function affects system performance, slows down the service running speed, and
occupies disk space. Do not enable the function when the system is in commercial use. If you
want to enable this function to locate a fault, you must have your supervisor's permission. After
locating the fault, disable the function as soon as possible.
You can run the release_on command to stop DEBUG. For details, see the ~ismpmml/
sms_run/shell/env.rc file.
The log file of the MML server has two types: running log and operation log.
l The running log is in ~ismpmml/sms_run/log/runlog. It records the running of the MML

server.
For the fault of the database agency, see the sms_runlog_(101)sms_db_agent.log file.
l The operation log is in ~ismpmml/sms_run/log/oplog. It records the operations of the
operator.

AAA
For the failed operation in the WMAS menu, see the sms_oplog_internal.log file.
4.3 Typical Cases
4.3.1 Billing System Fails to Define a Subscriber

The following section considers the failure in defining a subscriber in the CDMA service as an
example.
Symptom
The operator fails to define a subscriber through the billing system after the MML server
interconnects with the billing system.
Alarm
The ~ismpmml/sms_run/log/oplog/sms_oplog_internal.log file records the following alarm:
[]2006-07-15 00:17:29[]4[]Result[]17[]cdmaadm[]internal[]2[]Executing command
failed[]17[]ADD C270 ACCOUNT[]RETN=1010, DESC="Sorry, the subscriber already
exists.";
Cause
According to the returned log, it can be concluded that the subscriber data already exists.
Solution
1. In the subscriber basic table of the database, check the state of the subscriber with a duplicate
name.
2. If the subscriber is in to-be-deleted state, you can infer that the subscriber is deleted, but
the subscriber data is still stored.
Delete the subscriber again in the WMAS. Select not to store the subscriber data.
3. If the subscriber is in normal state, you can infer that the subscriber name is valid and may
be in use. Add the subscriber with a new name.
4.3.2 Unlocking the Operator root
Symptom
The account of the operator root in the WMAS system is locked.
Alarm
None.
Cause
The root operator of WMAS system is locked.

AAA
Solution
1. Modify the operator status field.
Run the following command to change the value of status to 0 in the record of the root
operator in the operator database table.
update operator set status="0" where userid="root"
2. Restart the sms_secu_serv process.

Log in as user ismpmml and run the stop.sh secu command to terminate the
sms_secu_serv process. After a while, the system starts the sms_secu_serv process
automatically.
3. Log in to the system as the operator root to check whether the fault is rectified.

AAA
Troubleshooting - Linux 5 Troubleshooting the WMAS
5 Troubleshooting the WMAS
About This Chapter
5.1 Introduction
5.2 Solution
5.3 Typical Cases

AAA
5.1 Introduction
The WMAS provides a Web-based operation management interface for the carrier and the agent.
The carrier and the agent can visit the WMAS with the browser and manage the cross-area
service.
The fault of the WMAS system concerns the display and operation of the GUI window.
5.2 Solution
Check the Process

Log in to the system as operator webadm. Check whether the WMAS process is normal. Run
the p command to query. The result must contain one java process.
Check the Port

Run the following command to check the port:
netstat -an|grep port number
Check whether port 8443are normal. If the ports are in the LISTEN status, it indicates that the
port is connected properly.
Check the Log

To check the log, do as follows:
l Query all the operation records in the *opl.log file in ~webadm/apps/WEB-INF/logs/.

l Query all the running records in the *run.log file in ~webadm/apps/WEB-INF/logs/.
l Query the sms_oplog_internal.log file in ~ismpmml/sms_run/log/oplog to locate the
detailed causes if many operation failures occur.
Check the Database

For Oracle database, the methods are as follows:
check whether the status of the instances sysdb and billdb is normal.
NOTE
For the detailed operation procedures, refer to the HUAWEI AAA Routine Maintenance.
5.3 Typical Cases
5.3.1 Defining the Subscriber Fails

AAA
Symptom
The WMAS administrator cdmaadm fails to define a subscriber.
Alarm
l The WMAS system prompts the following information:
Sorry, the subscriber already exists.
l The following alarm information is displayed in the websmap_opl.log file under the
~websmap/apps/WEB-INF/logs directory:
[Time] 00:17:29 July 15, 2006 733[LogID]04[LogType]MML[User Name]cdmaadm
[SessionID]05317713227651599695 [ACK]{TRANCONTROL=TXEND, TRANID=19,
DESC="Sorry, the subscriber already exists.", OPER=ADD C270 ACCOUNT,
SESSID=18, SESSCONTROL=DLGCON, RETN=1010}
Cause
The subscriber already exists.
Solution
1. Log in to the WMAS system as administrator cdmaadm.
2. When defining a subscriber, ensure that the user name is unique.
5.3.2 Logging In to the WMAS Fails
Symptom
A WMAS operator fails to log in to the system.
Alarm
The system displays "Reached the maximum number of operator logins."
Cause
The operator has consecutively entered the wrong password three times. Thus, the system locks
the operator.
Solution
1. Log in to the WMAS system as the operator of a higher level.
NOTE
If an operator of the normal administrator level is locked, the operator needs to be unlocked by an operator
of the service administrator level (such as cdmaadm).
If an operator of the service administrator level is locked, the operator needs to be unlocked by an operator
of the carrier system administrator level (such as servadm).
2. Unlock the operator.

AAA
Troubleshooting - Linux 6 Troubleshooting the Billproc
6 Troubleshooting the Billproc
About This Chapter
6.1 Introduction
6.2 Solution
6.3 Typical Cases

AAA
6.1 Introduction
The Billproc provides the functions of the bill delivery, input, statistics, and settlement.
The Billproc gets the bills from the iSCC1 and iSCC2 directory. The Billproc also generates the
settlement result according to the bill statistics and settlement strategy, and save them to the
database.
The fault of the Billproc concerns the processing of the bills.
6.2 Solution
Check the Log
The Billproc faults can be located through logs. The Billproc logs include the running logs and
error logs, as listed in Table 6-1.
Table 6-1 Billproc logs
Log Description
billget.log Records the following:

l Parameters read from the configuration file
l Information about deleting the temporary backup files
l Start time and end time of processing and transmitting the bill
billprocXX.log Records the following:

l Information about loading the bill into the database and statistics
billput.log Records the following:

l Information about uploading the oss bills
AccessFailRecordXX Records the following:

.log l Parameters read from the configuration file
l Information about importing online failure logs
AccessFailRecordCvt Records exceptions and errors during the running of the

.log AccessFailRecordCvt process.
XXbillcvt.log Records exceptions and errors during the running of the billcvt
process.
billgeterr.log Records exceptions and errors during the running of the billget
process.

AAA
Log Description
billprocerrXX.log Records exceptions and causes during the running of the billproc
process.
billputerr.log Records exceptions and errors during the running of the billput
process.
The causes of the Billproc faults can be found from the preceding logs. For example, if the logs
contain the information about the database operation faults, you can check the SQL statement
and error number to locate the fault.
The Billproc processing mechanism ensures that the bill cannot be lost. The bill that is not loaded
to the database is saved in the ~billproc/record/bad directory. If the Billproc fails to get the
bill, the related bill file is still saved in the original bill directory.
Check the Error Code

For Oracle database, do as follows:
Log in to the system as user billproc. Then run the oerr ora error code command to view the
details about the error code.
6.3 Typical Cases
6.3.1 Loading the Bill into Database Fails Because of Insufficient

Data Space
Symptom
The bills cannot be loaded into the database.
Alarm
Check the ~billproc/record/entry directory. A bill file always exists. For example, the
ps1_20061111_10010_00000001.default.txt bill file. View the contents of the bill and no error
information is found. The contents of the bill file are as follows:
ps1_20061111_10010_00000001.default.txt:
2|103|10|ddd|0|1|7|210|10.164.73.119||0|186|186|544|544|0|280||0.0.0.1|84565374|
169||1010|zyf access SV3|||3|8|1|20061111235722|20061111235722|20061111235822|60|
1|||0|0|02525065103-1185897562-102||0|
ps1_20061111_10010_00000001.default.txt:
3|103|10|ddd|0|1|7|210|10.164.73.119||0|186|186|544|544|0|280||0.0.0.1|84565374|
169||1010|zyf access SV3|||1|8|1|20061111235722|20061111235822|20061111235900|38|
3|||0|0|02525065103-1185897562-102||0|
ps1_20061111_10010_00000001.default.txt:
4|103|10|ddd|0|1|7|210|10.164.73.119||0|64|64|187|187|0|280||0.0.0.1|84565374|

AAA
169||1010|zyf access SV3|||2|8|1|20061111235722|20061111235900|20061111235913|13|

2||||0|02525065103-1185897562-102||0|
Cause
Consider the Oracle database as an example. Check whether the space of the bill database is
sufficient.
Log in to the system as user oracle. Then run the following commands:
setenv ORACLE_SID billdb
sqlplus "/ as sysdba"
select t.tablespace_name, round(sum(d.bytes/(1024*1024)),0) total_size, round(sum
(f.bytes)/(1024*1024)) as free_space, round(sum(f.bytes)/(1024*1024)*100/round(sum
(d.bytes/(1024*1024)),0)) as capacity from dba_tablespaces t, dba_data_files d,
dba_free_space f where t.tablespace_name = d.tablespace_name and t.tablespace_name
= f.tablespace_name group by t.tablespace_name;
Check the return result of the system. If the value in the free column that matches with the
database table name of RECORDDBS is 0 or approaching 0, it indicates that the database space
is insufficient.
Solution
Delete the expired bills.

AAA
Troubleshooting - Linux 7 Troubleshooting the Portal
7 Troubleshooting the Portal
About This Chapter
7.1 Overview
7.2 Typical Cases

AAA
7.1 Overview
Broadband users access the Internet on Portal in wireless mode. As a portal of the RADIUS
server, Portal cooperates with the RADIUS server to authenticate and authorize users.
Portal faults are associated with user authentication.
7.2 Typical Cases
7.2.1 Failed to Obtain an IP Address Due to Incorrect BRAS

Configuration
Symptom
When accessing WLAN on Portal, a user failed to obtain an IP address.
Check the portalserver.log file in $HOME/tomcat/logs for the keywords ipaddress not
found. If the keywords are found, the IP address cannot be obtained.
2011–11–14 12:34:47,273|DEBUG|BrasDatabase.java Line:64|[getBrasInfoByAddress]
fullAddress = null
2011–11–14 12:34:47,274|INFO|BrasDatabase.java Line:68|[getUserIpAddress]
fullAddress = null
2011–11–14 12:34:47,275|ERROR|LogoutServlet.java Line:76|[getBrasAddress]
ipaddress not found in t_bras_dhcpAddress = null
Cause
The possible causes are as follows:
l For Portal in earlier version (with the HSQLDB database), the IP address segment is
incomplete during the BRAS configuration.
l For Portal in earlier version (with the HSQLDB database), the local database is not restarted
after the BRAS configuration.
Solution
1. Log in to Portal as the portal user.
2. Access /export/home/portal/hsqldb and modify the bpss.script file.
vi bpss.script
3. Check whether one or more IP address segments that contain less than three digits exist.
l If yes, do as follows:
a. Add zeros to IP address segments that contain less than three digits.
An example is as follows:
INSERT INTO T_BRAS_DHCP VALUES
('1','BPSS_BRAS_1','255.255.255.255','0.0.0.0')
After the modification, the IP address is as follows:

AAA
INSERT INTO T_BRAS_DHCP VALUES

('1','BPSS_BRAS_1' ,'255.255.255.255','000.000.000.000' )
b. Restart the HSQLDB database.

Access /export/home/portal/hsqldb and run the following command to stop the
database:
./stop.sh
Run the following command to start the local database:

./startup.sh
l If the IP address segment format is correct, check whether the local database is restarted
after the BRAS configuration. If yes, restart the HSQLDB database. For details, see b
in 3.
7.2.2 Failed to Display the Portal Login Page Due to Overlarge Log
File or Connection Number
Symptom
A user cannot access the Portal login page.
Alarm
None
Cause
The possible causes are as follows:
l The number of connections exceeds the threshold.

l The log file is overlarge.
Solution
l If the Portal login page cannot display because the number of connections exceeds the
threshold, do as follows:
1. Log in to the Portal server as the root user.
2. Run the following command to modify the /etc/sysctl.conf file:
vi /etc/sysctl.conf
3. Set net.ipv4.ip_conntrack_max to 10000 (indicating that a maximum of 10000 TCP

connections can be established). If the /etc/sysctl.conf file does not contain this
parameter, add the following information:
net.ipv4.ip_conntrack_max = 10000
NOTE
If the log file contains TCP connection error information, set net.ipv4.ip_conntrack_max to
65536.
4. Set net.ipv4.ip_local_port_range to 1024 65000, indicating that the port number for
Portal to connect to external systems ranges from 1024 to 65000.
5. Add the following configuration items:
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_keepalive_time = 1800

AAA
net.ipv4.tcp_fin_timeout = 30
net.core.netdev_max_backlog =8096
The configuration items are described as follows:

– net.ipv4.tcp_tw_reuse: whether to enable the reuse function (0: disable; 1: enable).
The default value is 0.
– net.ipv4.tcp_tw_recycle: Whether to enable the function of fast recycling TIME-
WAIT sockets in TCP connections (0: disable; 1: enable). The default value is 0.
– net.ipv4.tcp_keepalive_time: Frequency for TCP to send a keepalive message, in
seconds. The default value 2 hours is changed to 30 minutes.
– net.ipv4.tcp_fin_timeout: If Portal requires sockets to be disabled, this parameter
specifies the interval for keeping the FIN-WAIT-2 state.
– net.core.netdev_max_backlog: length of the SYN queue. The default value is
1024, and the extended length is 8096. The extended SYN queue allows more
network connections that are waiting.
6. Run the following command to make the preceding modifications effective:
sysctl –p
l If the Portal login page cannot display because the log file is overlarge, do as follows:
1. Access /export/home/portal/tomcat/logs and back up the catalina.out file.
2. Delete the catalina.out file.
rm catalina.out
3. Create the catalina.out file.

touch catalina.out

AAA
Troubleshooting - Linux A Troubleshooting the Network
A Troubleshooting the Network
A.1 Overview
A.2 Solution
A.3 Typical Cases

AAA
A.1 Overview
The related network devices of the AAA include two LAN switches, two firewalls and mapping
access devices.
A.2 Solution
The network faults mainly focus on the firewalls and access devices, including the network block
and attack.
You are recommended to locate and rectify the fault with the engineer at the device side when
troubleshooting the network.
A.3 Typical Cases
A.3.1 Firewall not Supporting Session Update
Symptom
Previous versions of the firewall do not support the session automatic update. Thus after the
two-node switchover, the data package is sent to the original logic host rather than the new one.
Alarm
None.
Cause
The firewall does not support auto session update.
Solution
Enable the clear session function on the two-node cluster so that the firewall session can be
automatically cleaned upon two nodes switchover.
A.3.2 Description of Ports Mapping on Firewall
Symptom
l The user cannot pass the authentication.
l The user cannot log in to the portal.
Alarm
None.

AAA
Cause
The mapping between the ports on the firewall and the destination IP addresses is incorrect.
For example, if the port 1812 is mapped to the floating IP address of the ismswork resource, all
the authentication requests are sent to the ismswork resource for authentication. Therefore, the
user cannot pass the authentication.
Solution
Check the floating IP addresses of the isccwork and ismswork resources. Then modify the
mapping between the ports and the floating IP addresses according to Table A-1.
Table A-1 Description of the ports
Port Port Type Description

Numbe
r
isccwork port informaiton
1812 UDP Broadband RADIUS authentication port
1813 UDP Broadband RADIUS charging port
19999 TCP Billing system connection port
443 TCP User Portal HTTPS port
8443 TCP WMAS HTTPS port
9443 TCP Administration Portal HTTPS port
A.3.3 Subscriber Fails to Go Online After Switchover of the Two-

Node Cluster
Symptom
The online subscriber cannot go offline after the switchover of the two-node cluster. Afterwards,
the subscriber cannot be authenticated successfully.
Alarm
None.
Cause
The subscriber does not go offline normally after the switchover of the two-node cluster. Thus,
the account is still in use. The AAA limits the number of concurrent users of an account.
Therefore, the user cannot log in to the network using this account.

AAA
Solution
There are two methods, manual reset and auto reset.
Manually reset the abnormal subscriber in batches with the function of resetting by device
provided by the WMAS. In addition, the AAA can reset the subscribers automatically.
Configuration parameters related to the automatic reset are shown as follows:
l Charge Period: If the device supports the message of interim charge, the AAA automatically
resets when it detects that the message is timeout. That is, the AAA receives no message
of interim charge 30 minutes after the charge period.
l Maximum Session Duration: If the device does not support the message of interim charge,
the AAA executes the automatic reset when the subscriber online duration exceeds the
value of the parameter.
A.3.4 Subscriber Fails to Go Online Caused by Device Address

Change
Symptom
l Subscribers fail to go online.
l Use the setval tool to trace the subscriber who cannot log in. The U_man_debug.log file
records lots of Unknown IP information.
NOTICE
Alarm
None.
Cause
The device address is changed and the AAA does not modify the information synchronously.
Solution
Log in to the WMAS as servadm and modify the IP address configuration parameter of the
access device in the AAA system through the WMAS.

AAA
A.3.5 Subscriber Fails to Go Online Caused by Different Shared

Keys Between Device and the AAA
Symptom
l Subscribers fail to go online.
l Use the setval tool to trace the subscriber who cannot log in. Then the debug log
U_scfn_debug.log records many prompts containing "Wrong Password."
NOTICE
Alarm
None.
Cause
The shared key is different from the shared key configured in the AAA.
Solution
Log in to the WMAS as servadm and modify the shared key configuration parameter of the
access device in the AAA system to the same shared key configured in the access device.

AAA
Troubleshooting - Linux B The setval Command Reference
B The setval Command Reference
B.1 setval Commands
B.2 Module List

AAA
B.1 setval Commands

When you run the setval command, the system displays the help information and the command
syntax.
Table B-1 describes the commands and parameters.
NOTE
The symbol "/" indicates that either one can be used. For example, setval trace 1 equals to setval t 1.
Table B-1 setval commands and parameters
Statement Description
setval 0 0 Disable the message tracing function.
setval all/ALL Set to print all types of messages.
setval trace/t [0,1,2] Set the message printing type.0: Print the message.1:
Print the messages of the session numbers that you want
to trace.2: Print all the messages.
setval dialogue/dlg [1...10000] Set to print the message of a certain session. After the
iSCC2 restarts, the session number starts from 1 and
increases 1 for each call. When reaching
maxDlgIDOfSCP of SCPCFG.INI, the session number
starts from 1 again.
l all: Print the messages of all the session numbers.
l none: Do not print the message of any session
number.
setval level/l [0,1] Set the message printing level.0: Print brief messages.
1: Print detailed messages.
setval mode/m [0,1] Set the message printing destination.0: Print to the
screen.1: Print to the printing information file in the
log directory.
setval interval/i [1...n] Set the interval for printing the statistics messages. The
minimum interval is one minute.
setval clear/c [1,2,3,4,5,6] Clear the counter of the statistics messages to zero.
setval query/q [0,1,2] Query the setting of setval.
setval help/h Display the Help information of setval, mainly the

traceable level-1 module number and level-2 module
number.
setval [operationType] Set the traced module number. See B.2 Module List.
[operationCode]

AAA
Statement Description
setval [operationType] Modify the tracing setting of the module. See B.2
[operationCode] [value] Module List.
setval version/v Display the version information about each module of

the iSCP.
setval printTimes/pt [times] Number of lines to be printed. The part exceeding the
number of lines is not printed.Value range: 0 to 231 - 1
setval printSeconds/ps [seconds] Duration for printing the information, in seconds. After
the set duration is reached, the printing stops
automatically.Value range: 0 to 231 - 1
setval username/u [username] Trace a certain user name.none: Cancel tracing the user
name or do not trace the user name.
NOTE
l The user name and called number cannot be traced at the
same time.
l This command is mainly used in tracing the messages of a
broadband AAA user.
setval callingnumber/ca Trace a certain called number.none: Do not trace the

[callingnumber] called number.
NOTE
l The user name and called number cannot be traced at the
same time.
l This command is mainly used in tracing the messages of a
CDMA or WCDMA user.
setval printdlg/pd [username/ Print the information about the session and automaton.
CallingNumber]
setval multidlgID/md number Trace sessions of a single user. In the command,

number indicates the number of online sessions to be
traced in addition to those of a single user and depends
on the setval multidlgID/md command. The maximum
value of number is 10. This command is encapsulated
in the tracebycallingstationid.sh and
tracebyusername.sh scripts.
NOTE
The tracebycallingstationid.sh and tracebyusername.sh
scripts are saved in the /export/home/iscc2/bin directory.
B.2 Module List

When you run the setval h or setval help command, the system displays the Help information
and lists the level-1 module numbers and level-2 module numbers that can be set.
The level-1 module refers to operationType in setval [operationType] [operationCode].

AAA
Table B-2lists the mapping between the level-1 module number and the module name.
Table B-2 List of level-1 modules
No. Module
1 SCF
2 SDF
3 Manager
4 Server
5 Performance
6 SQL
The level-2 module refers to operationCode in setval [operationType] [operationCode].
The mapping between the level-2 module number and the module name is omitted. Refer to the
Help of the setval h command.

AAA
Troubleshooting - Linux C Acronyms and Abbreviations
C Acronyms and Abbreviations
AAA Authentication, Authorization and Accounting
API Application Program Interface
BRAS Broadband Remote Access Server
DB DataBase
GGSN Gateway GPRS Support Node
ID Identity
IP Internet Protocol
iSCC Integrated Service Control Center
iSMS Integrated Service Management System
LAN Local Area Network

AAA
Troubleshooting - Linux C Acronyms and Abbreviations
MML Man Machine Language
ODBC Open Database Connectivity
PC Personal Computer
PDSN Packet Data Service Node
SMP Service Management Point
TC Terminal Connector
WMAS Web Management Access Server

AAA
Troubleshooting - Linux Index
Index
A mml server
definition, 14
alarm information Introduction, 14
definition, 4 log, 14
Analyzing the Log Information, 5 port, 14
Process, 14
B
Bill
N
Insufficient Data Space, 22 network problem
Billproc cluster switchover, 30
functions, 21 device address, 31
processing mechanism, 22 firewall session, 29
shared keys, 32
C
Classification
O
Troubleshooting, 2 Original information
definition, 4
F
Flow
R
Troubleshooting, 2 root
unlock, 15
I
Indicator Status, 4
S
iscc1 setval, 10
Configuration File, 7 command, 34
functions, 7 module list, 35
Log, 7
Process, 7 T
iscc2
configuration file, 10 Tracing and Analyzing Messages, 5
function, 9 Troubleshooting
Introduction, 9 Analyzing Faults , 4
log, 9 Classification, 2
process, 9 Collecting Information , 3
Tracing, 10 Rectifying Faults , 4
M W
manual reset abnormal subscribers, 31 WMAS

AAA
Troubleshooting - Linux Index
log in fails, 19
unlock operator, 19
wmas
Defining the Subscriber Fails, 18


Troubleshooting 10 (Linux)

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Troubleshooting 10 (Linux)

Uploaded by

Copyright:

Available Formats

AAA

HUAWEI TECHNOLOGIES CO., LTD.

Trademarks and Permissions

Huawei Technologies Co., Ltd.

Issue 10 (2014-03-14) Huawei Proprietary and Confidential i

About This Document

System Maintenance Engineer

Indicates an imminently hazardous situation which, if not

Indicates a potentially hazardous situation which, if not

Indicates a potentially hazardous situation which, if not

Issue 10 (2014-03-14) Huawei Proprietary and Confidential ii

Indicates a potentially hazardous situation which, if not

NOTE Calls attention to important information, best practices and

Updates in Issue 10 (2014-03-14)

Updates in Issue 09 (2014-01-28)

Updates in Issue 08 (2010-03-31)

Issue 10 (2014-03-14) Huawei Proprietary and Confidential iii

The product version V200R002 is changed to R002C10.

Updates in Issue 07 (2009-09-30)

The product name infoX AAA is changed to HUAWEI AAA.

Updates in Issue 06 (2009-01-09)

The product name infoX-AAA is changed to infoX AAA.

Updates in Issue 05 (2008-12-10)

Modify the test questions.

Updates in Issue 04 (2008-11-15)

Modify the test questions.

Updates in Issue 03 (2007-09-30)

Updates in Issue 02 (2007-07-31)

"Appendix C The setval Command Reference" is added.

Updates in Issue 01 (2007-01-30)

Issue 10 (2014-03-14) Huawei Proprietary and Confidential iv

About This Document.....................................................................................................................ii

2 Troubleshooting the iSCC1.........................................................................................................6

3 Troubleshooting the iSCC2.........................................................................................................8

4 Troubleshooting the MML Server...........................................................................................13

5 Troubleshooting the WMAS.....................................................................................................17

Issue 10 (2014-03-14) Huawei Proprietary and Confidential v

5.1 Introduction .................................................................................................................................................................18

6 Troubleshooting the Billproc....................................................................................................20

7 Troubleshooting the Portal........................................................................................................24

A Troubleshooting the Network.................................................................................................28

B The setval Command Reference..............................................................................................33

C Acronyms and Abbreviations..................................................................................................37

Issue 10 (2014-03-14) Huawei Proprietary and Confidential vi

About This Chapter

1.1 Classification of Troubleshooting

1.2 Flow for Troubleshooting

1.3 Fault Analysis Methods

Issue 10 (2014-03-14) Huawei Proprietary and Confidential 1

1.1 Classification of Troubleshooting

1.2 Flow for Troubleshooting

1.2.1 Flow for Troubleshooting

Table 1-1 Troubleshooting flow

Number Task Action

1 Back up the database Prepare before the troubleshooting.

2 Back up the logs such as the Prepare before the troubleshooting.

Issue 10 (2014-03-14) Huawei Proprietary and Confidential 2

Number Task Action

1.2.2 Collecting Information

Information Collection Approaches

l Faults reported by users or the customer service center

Information Collection Methods

The skills are as follows:

Issue 10 (2014-03-14) Huawei Proprietary and Confidential 3