You are on page 1of 45

OSSW1274

NetAct OSS4.2

Troubleshooting 3GPP CORBA


Northbound Interface Problems

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

1 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

The information in this document is subject to change without notice and describes only the
product defined in the introduction of this documentation. This documentation is intended for the
use of Nokia Siemens Networks customers only for the purposes of the agreement under which
the document is submitted, and no part of it may be used, reproduced, modified or transmitted in
any form or means without the prior written permission of Nokia Siemens Networks. The
documentation has been prepared to be used by professional and properly trained personnel,
and the customer assumes full responsibility when using it. Nokia Siemens Networks welcomes
customer comments as part of the process of continuous development and improvement of the
documentation.
The information or statements given in this documentation concerning the suitability, capacity, or
performance of the mentioned hardware or software products are given as is and all liability
arising in connection with such hardware or software products shall be defined conclusively and
finally in a separate agreement between Nokia Siemens Networks and the customer. However,
Nokia Siemens Networks has made all reasonable efforts to ensure that the instructions
contained in the document are adequate and free of material errors and omissions. Nokia
Siemens Networks will, if deemed necessary by Nokia Siemens Networks, explain issues which
may not be covered by the document.
Nokia Siemens Networks will correct errors in this documentation as soon as possible. IN NO
EVENT WILL NOKIA SIEMENS NETWORKS BE LIABLE FOR ERRORS IN THIS
DOCUMENTATION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL,
DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT
NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS
OPPORTUNITY OR DATA, THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE
INFORMATION IN IT.
This documentation and the product it describes are considered protected by copyrights and
other intellectual property rights according to the applicable laws.
The wave logo is a trademark of Nokia Siemens Networks Oy. Nokia is a registered trademark of
Nokia Corporation. Siemens is a registered trademark of Siemens AG.
Other product names mentioned in this document may be trademarks of their respective owners,
and they are mentioned for identification purposes only.
Copyright Nokia Siemens Networks 2008. All rights reserved.

2 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

About this document

About this document


Troubleshooting 3GPP CORBA Northbound Interface Problems provides
you guidelines on how to solve the most common operational problems in
UMANBI fault management. This document can only provide the basic
guidelines, because some operating problems are fairly complex, often
depending on the network environment. This document assumes that
Nokia NetAct has been operating normally, and no problems occurred
during or after the integration of a new network element, or during
commissioning or upgrading.

1.1

NetAct compatibility and capacity information


For information on NetAct system and capacity, and the compatibility
between NetAct and network element releases, see the NetAct
Compatibility and Capacity document.

1.2

Terms
The following table presents some terms, abbreviations, and concepts. For
information on abbreviations that are not listed in this table, see Glossary.

Table 1.

DN05171601
Issue 3-0 en

Terms and abbreviations

Terms or
abbreviation

Explanation

Basic CM

Basic Configuration Management

EP

Entry Point

Extended TCL

Extended Trader Constraint Language

FM

Fault Management

FT

File Transfer

# Nokia Siemens Networks

3 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

Table 1.

Terms and abbreviations (cont.)

Terms or
abbreviation

Explanation

IRP

Integration Reference Point

IOR

Interoperable Object Reference


An IOR contains the information required for a client ORB
to connect to a CORBA object or servant

Kernel CM

Kernel Configuration Management

ORB

Object Request Broker


The CORBA ORB provides allows communication
between client requests and target objects.

1.3

PM

Performance Management

3GPP IRP External


Manager

Generic reference used throughout this document to refer


to the 3GPP IRP Manager in a multivendor network
system where the IRP Manager is not provided by Nokia.

Environment variables
This document assumes that the following environment variable has the
value indicated below:

Table 2.

Environment variable used in this document

Variable

Value

$OMCLOGDIR

/var/opt/nokianms/log

$ETCROOT

/etc/opt/nokiaoss

All log files are written to $omc/opt/*.

4 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

About this document

1.4

DN05171601
Issue 3-0 en

How to use this document


1.

This document is divided into the following categories:


.
Troubleshooting Entry Point IRP problems
.
Troubleshooting Alarm IRP problems
.
Troubleshooting Topology IRP problems
.
Troubleshooting Notification IRP problems
.
Troubleshooting Communication Surveillance IRP problems
.
Troubleshooting PM IRP problems
.
Troubleshooting File Transfer IRP problems

2.

In case of a problem, proceed to the appropriate chapter depending


on the problem you are experiencing. The tables in each chapter
offer you instructions on how to solve your problems.

3.

Follow the appropriate checkup and recovery procedures to solve


the problem.

4.

If the problem persists, contact your local Nokia Technical Support


Personnel. For further information, see the Helpdesk Rulebook.

# Nokia Siemens Networks

5 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

6 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting 3GPP CORBA Northbound Interface

Troubleshooting 3GPP CORBA


Northbound Interface
This chapter addresses possible problems with the NetAct 3GPP CORBA
Northbound Interface implementation in your system.
Before you start troubleshooting ensure that the Nokia NetAct system is up
and running.

Note
Most of the troubleshooting tasks are done on a NetAct server on
which the umanbi system component processes run. You need to log in
as user, omc.

2.1

About 3GPP CORBA Northbound Interface areas


The potential problems and recommended solutions are described in the
following chapters. The possible problems can be classified under the
following categories:

DN05171601
Issue 3-0 en

Entry Point IRP problems

Alarm IRP problems

Topology IRP problems

Notification IRP problems

Communication Surveillance IRP problems

PM IRP problems

File Transfer IRP problems

# Nokia Siemens Networks

7 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

8 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting CORBA Northbound Interface problems

Troubleshooting CORBA Northbound


Interface problems
The problems covered in this chapter include instructions on
troubleshooting Integration Reference Points (IRPs). These IRPs are listed
and described in this chapter.
The various IRPs in the 3GPP specification are co-dependent. For
example, access to the Alarm IRP CORBA object is enabled through the
Entry Point IRP. When you start troubleshooting an IRP problem make
sure that you identify the root IRP that is the cause of the problem. To do
this, check the log file or the trace file of the external IRP manager. Locate
the first problematic IRP and resolve this problem first.
For details on troubleshooting a specific IRP refer to the appropriate
sections in this chapter.

Note
Before you start any troubleshooting activity note that some of the tasks
need to be executed on the external IRP manager and some
information must be derived from the manager. Ensure that during
troubleshooting that the 3GPP External Manager documentation is
available and that you are able to execute tasks through the 3GPP
External Manager.

To obtain the correct server name for Entry Point IRP


components
1.

Start a system command prompt window and enter the following


command:
omc% ldapacmx.pl -sgPkgHost osscore2

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

9 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

Note
Make sure that you are logged into the server running osscore2.

If the 3GPP External Manager does not contain information on defective


IRPs, check the IRPs in the following order of priority:
1.

Entry Point IRP

2.

Basic CM

3.

Kernel CM

4.

Alarm IRP (operations)

5.

Alarm IRP (notifications)

6.

Notification IRP

7.

Communication Surveillance IRP

Note
Some of the IRPs are optional features. Therefore, some IRPs may not
be available on your system if the IRP licences are not purchased and
activated.

3.1

Entry Point IRP problems


As mentioned earlier in this chapter, the Entry Point IRP provides access
to other IRPs. When this IRP fails to function as specified, the 3GPP
external manager cannot connect to other IRPs.

3.1.1

Resolving Entry Point IRP problems


To resolve Entry Point IRP problems check the following in the order listed
below:

10 (45)

1.

Check the log file: werlogA5A9.log

2.

Check if the c3gcom process is running

3.

Check if there are DCN problems

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting CORBA Northbound Interface problems

4.

Check the IOR accessibility

5.

Check hostnames and IP addresses

Check the log file


1.

Open the werlogA5A9.log log file and check if any problems have
been logged in the file.

omc% ls $OMCLOGDIR/werlogA5A9.log

If there are no problems reported in the log file, continue with


checking for DCN problems.
2.

Read the log file, locate the problem and correct the existing
problem.

Check the c3gcom process


1.

Check that the c3gcom process is running. To do this, enter the


following command:
omc% mx grep 3gcom

If the process is running, check the IOR accessibility.


2.

If the c3gcomm process is not running, stop and restart all c3g
processes with the Nokia Process Supervision and Recovery
application (wpmanamx) running. To do this, enter the following
command:

omc% mx kill -HUP ls/var/opt/nokianms/ref/*cor2.c3gcommx* cut -d .-f3

Check the DCN problems


1.

Check if there are DCN problems. To do this, see section IP DCN


Problems in Troubleshooting System Management Problems and
refer to the relevant section in the 3GPP External Manager
documentation, which describes setting up DCN.

Check the IOR accessibility


1.

Check if the IOR is accessible via telnet. To do this open a telnet


connection from the 3GPP External Manager to the NetAct
osscore2 package with the port number 10115. Enter the following
command:
omc% telnet
omc% telnet> open <to> <IP address of osscore2> 10115

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

11 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

a.

Once you are connected to the correct osscore package,


enter the following command:
% GET/bin/get_nbif_ior

b.

If the command you entered at the command prompt returns


an IOR string, the IOR is accessible. The connection is closed
automatically.
Continue with the steps to resolve hostnames, for instructions, see
Check the IORs and resolve hostnames.
2.

If the IOR is not accessible, please check the network connection


and firewall settings.

Check the IORs and resolve hostnames


1.

Check if the IORs can be resolved on the 3GPP External Manager.


Write the IOR of the naming context and the IOR of the Entry Point
IRP to temporary files. To do this, enter the following commands:

omc% nsconfmx cat NokiaOSS/local/services/3GPP/rel_6/EPIRP > /tmp/epirp.ior


omc% nsconfmx cat NokiaOSS/local/services/3GPP/rel_6 > /tmp/nc.ior

2.

Locate the hostnames in the IOR files that you saved in your tmp
directory.
omc% /opt/tao/bin/catior -f /tmp/epirp.ior
omc% /opt/tao/bin/catior -f /tmp/nc.ior

3.

Locate the line in your concatenated tmp file that looks similar to this:
Hostname: r020mes.r020.netact.sutelab.dus

4.

Resolve the hostnames for both the Entry Point IRP and the naming
context on the external manager. To do this, enter the following
command:

nslookup <epirp_hostname> nslookup naming_context_hostname>

The hostnames can be resolved if the system output of the


nslookup for both the 3GPP External Manager and the naming
context looks similar to the following:
Trying DNS
looking up FILES nslookup Name Server: r020mes.r020.netact.sutelab.dus Address:
10.80.7.19

5.

12 (45)

If the IP-address cannot be resolved, set up the IP mapping of the


host names on the 3GPP External Manager.
a.
First obtain the hostnames and the IP address using the
nslookup command. For instructions, see step 4.
b.
Follow the instructions in your 3GPP External Manager
documentation on mapping the hostnames of the IORs on to
the 3GPP External Manager.

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting CORBA Northbound Interface problems

If these troubleshooting tasks do not address your problem, contact your


local Nokia Technical Support Personnel.

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

13 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

14 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting Alarm IRP problems

Troubleshooting Alarm IRP problems


The problems covered in this chapter include instructions on
troubleshooting Alarm Integration Reference Point (Alarm IRP) problems.

4.1

Alarm IRP problems (operations)


Note
The Alarm IRP is an optional feature. It is only enabled in your system if
the FM CORBA 6 NBIF/Northbound Interfaces/UMA/3GPP Northbound
Interface: Fault Management feature is purchased, and licences are
installed and activated.
If you do not have this feature installed in your system, skip this section
and continue with Troubleshooting Performance Management
Problems.

This section addresses possible problems that can occur for Alarm IRP
(operations) and troubleshooting actions to resolve these problems.

4.1.1

Resolving Alarm IRP problems (operations)


To resolve Alarm IRP problems check the following in the order listed
below:

DN05171601
Issue 3-0 en

1.

Check licence availability

2.

Check for iterator timeout

3.

Check the log file

# Nokia Siemens Networks

15 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

4.

Check if FM processes are running

5.

Check the alarm synchronisation and alarm states

Check licence availability


As mentioned in the note in section 4.1 Alarm IRP (operations) problems
1.

To check that the FM_CORBA_6_NBIF licence is available, enter


the following command:
omc% grep FM_CORBA_6_NBIF $LM_LICENCE_FILE grep -v #

2.

Correct the licence information, if necessary. Obtain the correct


licence files from your local Nokia Technical Support personnel.

3.

Follow the instructions in the configuration guide to activate and


configure the Alarm IRP feature.

Check for iterator timeout


1.

Open the werlogA5BE.log file. To locate the file, enter the following
command:
%omc ls $OMCLOGDIR/werlogA5BE.log

Note
The werlog numbers are:

2.

werlog A5BE.log: Alarm IRP operations

werlog A5BD.log: Alarm IRP notifications

werlog A5BF.log: Basic CMIRP (topology operations)

werlog A5C0.log: Kernel CMIRP (topology notifications)

Open the file in a text editor of your choice and locate this line:
Iterator timeout occurred

If you do not see this line, continue with the task Check problems in
log file.
3.

16 (45)

Correct the timeout period for the upload iterator.

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting Alarm IRP problems

4.

Note
The error message in the upload iterator timeout shows that the 3GPP
External Manager has not called the iterator in the configured timeframe in order to get additional data. Check the behaviour of your 3GPP
External Manager, why the time between two iterator calls is so long.
5.

If the 3GPP External Manager cannot process the results within the
time-frame of a single iteration, it is possible for you to increase the
time-frame. For instructions, refer to the 3GPP External Managers
configuration guide.

Check problem reports in log file


1.

Open the werlogA5BE.log file. To locate the file, enter the


following command:
%omc ls $OMCLOGDIR/werlogA5BE.log

2.

Open the file in a text editor of your choice or use the UNIX
concatenate function to read the file. Search for errors.

3.

If errors are found, correct them. If not, see task Check for iterator
timeout and continue with the Check if FM processes are running
procedure.

Check if FM processes are running


1.

Check that the 3GPP release 6 alarm synchronisation process,


c3gas6 is running. To do this, enter the following command:
omc% mx grep c3gas6

2.

If the process is running, continue with Check alarm synchronisation.

3.

Restart all c3g processes with the Nokia Process Supervision and
Recovery application (wpmanamx) running. To do this, enter the
following command:

omc% mx kill -HUP ls/var/opt/nokianms/ref/*cor2.c3gcommx* cut


-d .-f3

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

17 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

Check alarm synchronisation

4.2

1.

Trigger an alarm and start the alarm synchronisation process in your


3GPP External Manager.

2.

Check the alarm state in NetAct and the 3GPP. If the alarm state is
not identical in both systems, continue with the next step. If it is
working, continue with 4.2.1 Resolving Alarm IRP (notifications)
problems.

3.

Check if the alarm synchronisation process is running with a filter


which specified that it is to return everything, that is, the filter
definition has an empty string. To do this, start the alarm
synchronisation process with such a filter in the external IRP
manager.

4.

Now the alarm upload state in NetAct and the alarm state in the
3GPP External Manager must be similar. If they are not, you need to
correct the filter definition for the 3GPP external manager. The filter
definition must be specified in the Extended TCL syntax.

Alarm IRP problems (notifications)


Note
The Alarm IRP is an optional feature. It is only enabled in your system if
the FM CORBA 6 NBIF/Northbound Interfaces/UMA/3GPP Northbound
Interface: Fault Management feature is purchased, and licences are
installed and activated.
If you do not have this feature installed in your system, skip this section
and continue with Troubleshooting Performance Management
problems.

This section addresses possible problems that can occur for Alarm IRP
(notifications) and troubleshooting actions to resolve these problems.

4.2.1

Resolving Alarm IRP (notifications) problems


To resolve Alarm IRP problems, check the following in the order listed
below:

18 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting Alarm IRP problems

1.

Check licence availability.

2.

Check for iterator timeout.

3.

Check problem reports in log file.

4.

Check if FM processes are running.

5.

Enable tracing.

6.

Ensure that Network Watch (NWW) is running.

7.

Trigger an alarm, recreate the buffer and check the arcsysmx.cf for
umanbi_fm alarms.

For instructions on performing tasks 1 to 3, see section 4.1.1 Resolving


Alarm IRP (operations) problems. For instruction for task 4 to 7, refer to the
instructions in the following sub-sections.

Check if FM processes are running


1.

Check that the 3GPP release 6 alarm notification process, c3gaf6 is


running. To do this, enter the following command:
omc% mx grep c3gaf6

2.

If the process is running, check the log file for problems, see Check
problem reports in log file. If it is running, continue with the next step.

3.

Restart all c3g processes with the Nokia Process Supervision and
Recovery application (wpmanamx) running. To do this, enter the
following command:

omc% kill -HUP ls /var/opt/nokianms/ref/*cor2.c3gcommx* cut


-d "." -f3

Enable tracing in the c3gaf6mx process file


1.

Enable tracing for the 3GPP CORBA release 6 Alarm Notification


Forwarder process, c3gaf6mx

2.

Before you continue with the following steps, ensure that the Nokia
Network Watch (nww), system component is running. For
instructions, see About alarm pipes in Nokia NetAct Troubleshooting
Fault Management Problems.

3.

Check that the 3GPP CORBA release 6 Alarm Forwarding Service


process, c3gaf6mx is running. To do this, enter the following at the
command prompt:
omc% mx grep c3gaf6

4.

DN05171601
Issue 3-0 en

Enable tracing in c3gaf6mx.

# Nokia Siemens Networks

19 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

Open the corresponding (for c3gaf6mx) configuration file,


c3gconcmplamx_REL6.cf in a text editor, and edit the file.
You can find the file in /etc/opt/nokiaoss/uma/umanbi/common/
conf. Change the trace value for C3GAFS from False to True.
(C3GAFS
""
(TRACE
"FALSE")
(LEVEL
"default")
(TRACE_LIBRARY
"C3GAFS")
(REMOVE_TRACE_FILE
"False")

Note
For details on enabling tracing for the umanbi system component
processes, see section 5.1.1 Enabling tracing in Configuring NetAct for
3GPP CORBA Northbound Interfaces.
The trace is written to the c3gaf6mx.trc file in the $VARROOT/tmp/
trace directory.
5.

Kill the c3gas6mx process in order to trigger an alarm using the kill
command.
omc% kill - USR2 <PID of c3gas6mx>

20 (45)

6.

Wait a minute for the alarm to proceed through the alarm pipe.

7.

Check the trace file for the recently triggered alarm. If it is shown in
the trace without a failure report, continue with Troubleshooting
Notification IRP.

8.

Check the buffer for the Northbound Interface (nbi), release 6


subsystem component alarms.

9.

Recreate the buffer. For instructions, see section To recreate buffers


in Troubleshooting Fault Management Problems.

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting Topology problems

Troubleshooting Topology problems


The problems covered in this chapter include instructions on
troubleshooting network topology synchronisation problems.

5.1

Basic CM problems
Basic Configuration Management (Basic CM) problems refer to events
when the network topology is not synchronised with the existing topology
in NetAct.

5.1.1

Resolving Basic CM problems


To resolve Basic CM problems check the following in the order listed
below:
1.

Check licence availability

2.

Check for iterator timeout

3.

Check the log file

4.

Check if Basic CM processes are running

5.

Check the topology sychronisation

Check licence availability


As mentioned in the note in section 4.1 Alarm IRP (operations) problems
1.

To check that the BCM_CORBA_6_NBIF licence is available, enter


the following command:
omc% grep BCM_CORBA_6_NBIF $LM_licence_FILE grep
-v #

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

21 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

2.

Correct the Basic CM licence information, if necessary. Obtain the


correct licence files from your local Nokia Technical Support
personnel.

3.

Follow the instructions in the configuration file to activate and


configure the Alarm IRP feature.

Check for iterator timeout


1.

Open the werlogA5BF.log file. To locate the file, enter the following
command:
%omc ls $OMCLOGDIR/werlogA5BF.log

2.

Open the file in a text editor of your choice and locate this line:
Iterator timeout occurred

If you do not see this line, continue with the task Check problems in
log file.
3.

Correct the timeout period for the upload iterator.

4.

Note
The upload iterator timeout error message shows that the 3GPP
External Manager has not called the iterator in the configured timeframe in order to get additional data. Check the behaviour of your 3GPP
External Manager.
5.

If the 3GPP External Manager cannot process the results within the
time-frame of a single iteration, it is possible for you to increase the
time-frame. For instructions, refer to the 3GPP External Managers
configuration guide.

Check problem reports in log file


1.

Open the werlogA5BF.log file. To locate the file, enter the


following command:
$OMCLOGDIR/werlogA5BF.log

22 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting Topology problems

2.

Open the file in a text editor of your choice or use the UNIX
concatenate function to read the file. Search for errors.

3.

If errors are found, correct them. If not, see task Check for iterator
timeout and continue with the Check if BCM processes are running
procedure.

Check if BCM processes are running


1.

Check that the 3GPP release 6 topology synchronisation process,


c3gcs6 is running. To do this, enter the following command:
omc% mx grep c3gcs6

2.

If it is running, continue with Check topology synchronisation.

3.

Restart all c3g processes with the Nokia Process Supervision and
Recovery application (wpmanamx) running. To do this, enter the
following command:

omc% mx kill -HUP ls/var/opt/nokianms/ref/*cor2.c3gcommx* cut


-d .-f3

Check topology synchronisation

5.2

1.

Trigger a topology event and start the topology synchronisation


process in your 3GPP External Manager.

2.

Check the topology view in NetAct and the 3GPP. If the topology is
not identical in both systems, continue with the next step. If it is
working, continue with 4.2.1 Resolving Kernel CM problems.

3.

Check if the synchronisation process is running with a filter which


specified that it is to return everything, that is, the filter definition has
an empty string. To do this, start the topology synchronisation
process with such a filter in the 3GPP external manager.

4.

Now the topology view in NetAct and in the 3GPP external manager
must be similar. This shows that the fault lies with the use of a wrong
filter. You need to correct the filter definition for the 3GPP external
manager. The filter definition must be specified in the Extended TCL
syntax.

Kernel CM problems
When the Kernel Configuration Management fails, the 3GPP external
manager is not able to receive topology notifications and it is not informed
immediately of changes in the NetAct topology.

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

23 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

5.2.1

Resolving Kernel CM problems


To resolve Kernel CM problems, check the following in the order listed
below:
1.

Check licence availability.

2.

Check problem reports in log file.

3.

Check if Kernel CM processes are running.

4.

Enable tracing

5.

Ensure that Topology processes ( utp* processes ) are running.

6.

Trigger an event, recreate the buffer and check the arcsysmx.cf for
umanbi_fm alarms.

For instructions on performing tasks 1 to 3, see section 4.1.1 Resolving


Basic CM problems.For instruction for task 4 to 7, refer to the instructions
in the following sub-sections.

Check if Kernel CM processes are running


1.

Check that the 3GPP release 6 topology notification process,


c3gtf6 is running. To do this, enter the following command:
omc% mx grep c3gtf6

2.

If the process is running, check the log file for problems, see Check
problem reports in log file. If it is running, continue with the next step.

3.

Restart all c3g processes with the Nokia Process Supervision and
Recovery application (wpmanamx) running. To do this, enter the
following command:

omc% mx kill -HUP ls/var/opt/nokianms/ref/*cor2.c3gcommx* cut


-d .-f3

Enable tracing in the c3gtfs6mx process file


1.

Enable tracing for the 3GPP CORBA release 6 Kernel CM process,


c3gtf6mx

2.

Check that the 3GPP CORBA release 6 Kernel CM process,


c3gtf6mx is running. To do this, enter the following at the command
prompt:
omc% mx grep c3gtf6

3.

24 (45)

Enable tracing in c3gtf6mx.

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting Topology problems

Open the corresponding (for c3gtf6mx) configuration file,


c3gconcmplamx_REL6.cf in a text editor, and edit the file.
You can find the file in /etc/opt/nokiaoss/uma/umanbi/common/
conf. Change the trace value for C3GTFS from False to True.
(C3GTFS
""
(TRACE
"FALSE")
(LEVEL
"default")
(TRACE_LIBRARY
"C3GTFS")
(REMOVE_TRACE_FILE
"False")

Note
For details on enabling tracing for the umanbi system component
processes, see section 5.1.1 Enabling tracing in Configuring NetAct for
3GPP CORBA Northbound Interfaces.
The trace is written to the c3gtf6mx.trc file in the $VARROOT/tmp/
trace directory.
4.

Open nedmanmx for testing an RNC object.

Note
Delete the object after the testing.

DN05171601
Issue 3-0 en

5.

The object is visible after one minute in the trace file, if it is there,
continue with Troubleshooting Notification IRP.

6.

When the object is not visible, the problem is in the NetAct topology
system, check Troubleshooting System Management Problems to
correct the problem in the topology system.

# Nokia Siemens Networks

25 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

26 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting Notification IRP problems

Troubleshooting Notification IRP


problems
The 3GPP external manager uses the 3GPP CORBA Northbound
Interface Notification IRP to register itself to receive notifications which are
sent from other IRPs. When the Notification IRP fails, the 3GPP external
manager does not receive notifications about changes. In this case, the
3GPP external manager does not contain up-to-date information about
changes in the NetAct system.

6.1

Resolving Notification IRP problems


To resolve the Notification IRP problems check the following in the order
listed below:
1.

Check if the c3gcom process is running.

2.

Check the IOR accessibility

3.

Check that the Notification IRP version on the 3GPP External


Manager is different to the version in NetAct.

4.

Check for a subscription call with a defective filter

5.

Check for notifications using the ntfdumper.

Check the c3gcom process


1.

Check that the c3gcom process is running. To do this, enter the


following command:
omc% mx grep c3gcom

If the process is running, continue with the next task.

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

27 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

2.

If the c3gcomm process is not running, stop and restart all c3g
processes with the Nokia Process Supervision and Recovery
application (wpmanamx) running. To do this, enter the following
command:

omc% mx kill -HUP ls/var/opt/nokianms/ref/*cor2.c3gcommx* cut


-d .-f3

Check the IOR accessibility


1.

Check if the IOR of the Notification IRP is in the naming context. To


do this, enter the following command:

omc% nsconfmx list -r -l NokiaOSS/local/services/3GPP/rel_6 grep notification

2.

If the system returns this output:


omc% NokiaOSS/local/services/3GPP/rel_6/notificationirp

Check the IORs and resolve hostnames


1.

Resolve the hostnames for both the Notification IRP and the naming
context on the external system. To do this, enter the following
command:
nslookup <notificationirp_hostname>
nslookup <naming_context_hostname>

The hostnames can be resolved if the system output of the


nslookup for both the 3GPP External Manager and the naming
context looks similar to the following:

Trying DNS
looking up FILES
nslookup
Name Server: r020mes.r020.netact.sutelab.dus
Address: 10.80.7.19

In this case, continue with Check the Notification IRP versions.


2.

Set up the IP mapping of the host names on the 3GPP External


Manager.

Check the Notification IRP versions

28 (45)

1.

Check that the Notification IRP version on the 3GPP External


Manager is different to the version in NetAct.

2.

If there are no version problems continue with the next procedure.

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting Notification IRP problems

Check the log file


1.

Open the werlogA5A9.log log file and check if any problems have
been logged in the file.

omc% ls $OMCLOGDIR/werlogA5A9.log

If there are no problems reported in the log file, continue with the
step 3.
2.

Read the log file, locate the problem and correct the existing
problem.

Check the notification with ntfdumper


1.

Use the ntfdumper to view the nuntiumx notifications.


Start the ntfdumper:

omc% nsconfmx cat NokiaOSS/local/notifications/conAdmins/NotificationIRP:C3GCOM:


ManagerAdmin >/tmp/ior.txt
omc % netact_start.pl npf/npfnta
-- /opt/nokiaoss/npf/npfnta/lbin/ntfdmpmx -ca /tmp/ior.txt
-to 180

2.

Create an alarm event:


omc % mx grep c3gas6
Omc % kill -USR2 <pid c3gas6>

When the alarm notification is shown in the dumper, the Notification


IRP works, check the Consumer of the external manager.

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

29 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

30 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting Communication Surveillance problems

Troubleshooting Communication
Surveillance problems
To resolve Communication Surveillance (CS) IRP problems check the
following in the order listed below:
1.

Check if the c3gcom process is running, see Check the c3gcom


process in section Resolving Notification IRP problems.

2.

Check the IOR accessibility. For instructions, see Check IOR


accessibility in section Resolving Notification IRP problems.

3.

Check that the CS IRP version on the 3GPP External Manager is


different to the version in NetAct.

4.

Manually trigger a heartbeat from the 3GPP External Manager.

5.

Check the Notification IRP service.

For instructions, for tasks 1 and 2, refer to section Resolving Notification


IRP problems. For tasks 3 and 4 see the following instructions under the
procedure titles Check the IRP versions and manually trigger a heartbeat

Check the CS IRP Versions


1.

Check that the Communication IRP version on the 3GPP External


Manager is different to the version in NetAct.

2.

If there are no version problems, continue with the next procedure.

Manually trigger a heartbeat


1.

Manually trigger a heartbeat in the 3GPP external manager.

2.

Check the Notification IRP service.

Check the Notification IRP service


1.

DN05171601
Issue 3-0 en

For instructions, see chapter Troubleshooting Notification IRP


problems.

# Nokia Siemens Networks

31 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

32 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting PM process problems

Troubleshooting PM process problems


This chapter contains instructions on Troubleshooting PM processes that
are defective.
For details on troubleshooting a specific PM problem refer to the
appropriate sections in this chapter.
The possible problems that can occur with the 3GPP CORBA Northbound
are listed here:
.

PM processes are not running as specified, for more information,


see Problems with PM processes.

A PM measurement job cannot be created, for more information, see

A PM measurement job cannot be stopped.

A PM measurement job cannot be listed.

Note
Most of the troubleshooting tasks are done on a NetAct Server on which
the umanbi system component processes run. You need to log in as
omc user. You need to obtain the correct server name.

To obtain the correct server name


1.

Log in to the NetAct system and enter the following command:


omc% ldapacmx.pl -sgPkgHost rep

or
omc% ldapacmx.pl -sgPkgHost rep2

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

33 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

8.1

Problems with PM processes


There are two PM processes responsible for fetching files, converting
them to 3GPP compliant format, and transferring them to the export
directory from which the data can then be polled.
The Measurement Result Handler process, (c3gmrhmx) in the 3GPP
CORBA PM Northbound Interface is responsible for fetching performance
measurement result files in OMeS format from the NetAct Reporter PM
Pipe. It converts these files into 3GPP compliant format. Once this is done,
the 3GPP File Transfer Service process, (c3gfitmx) is notified. The
c3fitmx process then exports the converted files to an export directory.
This export directory can be polled regularly.
In some cases these PM result files are not fetched, converted and
exported successfully. This section contains instructions on
troubleshooting problems that affect these two processes:
.

c3gmrhmx

c3gfitmx

For more information on these two processes, see Exporting


Measurement Result Files in the 3GPP CORBA Northbound Interface
Technical Reference Guide.

Note
The 3GPP CORBA PM Northbound Interface contains an optional
package of features, for which the following licences are available:
3GPP_CORBA_PM_NBIF_BASIC, 3GPP_CORBA_PM_NBIF_UNCOND and
3GPP_CORBA_PM_NBIF_VS_DATA.

8.1.1

Resolving PM file processing problems


To resolve PM file processing problems, check the following in the order
listed below:

34 (45)

1.

Check PM licence availability

2.

Check the log file

3.

Check if PM processes are running

4.

Check the ListMeasurementJob operation

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting PM process problems

5.

Check the PM pipe filter and PM pipe output configuration files

6.

Check the working directories for the 3GPP File Transfer Service
process, (c3gfitmx)

Check PM licence availability


As mentioned in the note in section Problems with PM access, the 3GPP
CORBA PM Northbound Interface is an optional feature.
1.

To check that the 3GPP_CORBA_PM_BASIC and


3GPP_CORBA_PM_NBIF_UNCOND licences are available, enter the
following at the command prompt
omc% grep -i 3GPP_CORBA_PM_NBIF $LM_licence_FILE.
.

If the feature 3GPP_CORBA_PM_NBIF_BASIC is missing, PM


related processes will not start up.
If the feature 3GPP_CORBA_PM_NBIF_UNCOND is missing, PM
measurement result files are then only exported for
measurement jobs that are successfully created and
executed. The measurement job must be active for PM data to
be produced. If the licence is available, all available PM data is
exported via the 3GPP CORBA Northbound Interface.

2.

Correct the licence information, if necessary. Obtain the correct


licence files from your local Nokia Technical Support personnel.

3.

Follow the instructions in the configuration file to activate and


configure the two PM features.

Check problem reports in the log file


1.

Open the werlogA5A1.log and werlogA5A1.log log files. To locate


the file, enter the following at the command prompt:
$OMCLOGDIR/werlogA5A1.log
$OMCLOGDIR/werlogA5A2.log

DN05171601
Issue 3-0 en

2.

Open the file in a text editor of your choice or use the UNIX
concatenate function to read the file. Search for errors.

3.

If errors are found, correct them.

# Nokia Siemens Networks

35 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

Check if the relevant PM processes are running


1.

Check that c3gmrhmx and c3gfitmx are running. To do this, enter


the following command:
omc% ps -ef grep -v grep grep
c3gmrhmx
omc% ps -ef grep -v grep grep
c3gfitmx

2.

If it is not running, restart both processes. If it is running, continue


with the next task.

Check the notification service


1.

Check if the notification adaptor server process, nuntiumx is up and


running. To do this, enter the following command:
omc% ps -ef grep -v grep grep
nuntiumx

2.

Check if too many entries are returned by the operation: list available
files. If there are too many entries, select a shorter time interval. To
do this use the UNIX word count, list lines command:

omc% ls -ltr $VARROOT/uma/umanbi/pg_c3gcom/work/export wc


-l

Enable trace mode

36 (45)

1.

Enable tracing for the 3GPP CORBA PM c3gmrhmx and c3gfitmx


processes.

2.

Open the corresponding configuration file, pm1sebmx.cf,


pm2sebmxcd and c3gsebmx.cf, in a text editor such as, vi and edit
the configuration file. You can find these files here:
.
For pm1sebmx.cf see $ETCROOT/uma/umanbi/c3g_pm/conf/
.
.
For pm2sebmx.cf see $ETCROOT/uma/umanbi/c3g_pm/conf/
.
.
For pm2sebmx.cf see $ETCROOT/uma/umanbi/common/conf/
.
Change the trace value in each file from False to True.

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting PM process problems

Note
For details on enabling tracing for the umanbi system component
processes, see section 5.1.1 Enabling tracing in Configuring NetAct for
3GPP CORBA Northbound Interfaces.

Check the ListMeasurementJob operation

1.

Check that a measurement job has been created through the


ListMeasurementJob operation.

2.

Check the data flow for PM data for possible interruptions. For the
network element classes such as MMSC, MGW, SG3G, and MSC;
the measurements must be activated for each Network Element
(NE).

Note
For more information, refer to the data flow diagrams in sections
Overview of the Reporter and PM data flow, PM pipe in Nokia NetAct,
and Measurement processing data flow in Reporter and Performance
Management Technical Reference Guide.

Check the PM pipe filter and PM output configuration files

1.

Check the PM pipe filter configuration for possible filtering within the
PM core. To do this check the filter file. You can find the file in:
$ETCROOT/reporter/etload/conf/c3g_pm.filter

2.

Check the PM pipe output configuration files in the configuration


directory. You can find the file in:
$ETCROOT/reporter/etload/conf/c3g_pm.pipe.

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

37 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

3.

Compare the output of the PM pipe filter configuration file to the


configuration item OMES_FILE_DIRECTORY in the files pm1sebmx.
cf and pm2sebmx.cf

4.

Correct mismatches, if found by editing the files, pm1sebmx.cf and


pm2sebmx.cf. For more information on the PM pipe configuration,
see Reporter Technical Reference Guide.

Check the working directories for c3gfitmx


1.

Check the working directories in the NetAct system for the 3GPP
File Transfer Service process, c3gfitmx. To do this, enter the
following command
omc% find $VARROOT/uma/umanbi/pg_c3gcom/work/c3gfit

2.

If no transaction files are found, check for notification files in the


global communication directory. The default location for these
notification files is var/opt/nokiaoss/irp/c3gcom/comm .

3.

Check the c3gsembx.cf and pm1sebmx.cf configuration files for


consistency. The c3gsembx.cf configuration files can be found in
$ETCROOT/uma/umanbi/common/conf/ . The pm1sebmx.cf
configuration file can be found in $ETCROOT/uma/umanbi/c3g_pm/
conf.

4.

Check the werlogA5A3.log file in $OMCLOGDIR for problems with too


many files accumulating in the sub-directory which are deemed
active. Over a period of time, this over-accumulation may lead to
problems retrieving the data files from the file writers.
If you locate signs of this problem in the log file, check the
transmission services such as ftp/sftp/scp on the host server.

8.2

Problems with the PM measurement jobs


As mentioned in the CORBA 3GPP Northbound Interface Principles
document, with the PM IRP, a 3GPP IRP Manager can request the
creation of measurement jobs. These requests are validated by the PM
IRP server, stored persistently in a measurement job repository, and
forwarded to NEs which support measurement administration.
Furthermore, the 3GPP IRP Manager can list or stop active measurement
jobs. The following sections deal with problems that occur when
measurement jobs cannot be created, listed or stopped and the
troubleshooting tasks that you can carry out to resolve these problems.

38 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting PM process problems

8.2.1

Resolving problems with the measurement job creation function


To resolve PM IRP problems with creating measurement jobs, check the
following in the order listed below:
1.

Check PM licence availability

2.

Check the log file

3.

Check if the 3GPP PM IRP server is running and enable tracing

4.

Check the notification service

5.

Check the emugmamx processes

6.

Check managed objects Distinguished Names (DNs)

7.

Check that all counter names are legal

Check PM licence availability


As mentioned previously, 3GPP CORBA PM Northbound Interface is an
optional feature. Before you start troubleshooting tasks related to the PM
measurement jobs, ensure that you have the licence.
1.

To check that the licence


3GPP_CORBA_PM_NBIF_BASIC,3GPP_CORBA_PM_NBIF_UNCOND and
3GPP_CORBA_PM_NBIF_VS_DATA licence is available, enter the

following command:
omc% grep -i 3GPP_CORBA_PM_NBIF $LM_licence_FILE.

2.

Correct the licence information, if necessary. Obtain the correct


licence files from your local Nokia Technical Support personnel.

3.

Follow the instructions in the configuration file to activate and


configure the two PM features.

Check the log file


1.

Check for problem reports in the log files. Open the werlogA5A1.
log and werlogA5A2.log files. To locate the files, enter the
following at the command prompt:
omc% less $OMCLOGDIR/werlogA5A1.log
omc% less $OMCLOGDIR/werlogA5A2.log

2.

If errors are found, correct them.

Check that the 3GPP PM IRP server is running and enable


tracing

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

39 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

1.

To check that the 3GPP PM IRP server is running enter the following
command:
omc% ps -ef grep -v grep grep
c3gpirmx

2.

If it is not running, restart the server.

3.

Enable tracing for 3GPP PM IRP server process, c3gpirmx. To do


this, locate the corresponding configuration file, c3gsebmx.cf, in
etc/opt/nokiaoss/uma/umanbi/common/conf and set the trace
value from False to True.

Check the notification service


1.

Check if the notification adaptor server process, nuntiumx, is up and


running. To do this, enter the following command:
omc% ps -ef grep -v grep grep
numtiumx

2.

Restart the notification service, if it is not running.

Check the emugmamx process


1.

Check that the emugmamx process is running. The emugmamx process


is responsible for dispatching requests to network elements. To
check that emugmamx is running on the system server, do the
following:

2.

To obtain the correct NetAct system server name, enter the following
command:
omc% ldapacmx.pl -sgPkgHost osscore2

3.

Log in to the system server


omc% rlogin <name of system
server>

4.

Check that emugmamx is running:


omc& ps -ef grep -v grep grep
emugmamx

If it is not running, restart the process. If it is running, re-create the


previously unsuccessful measurement job.

Check the Distinguished Names of each Managed Object and


counters

40 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting PM process problems

1.

Check that all the Distinguished Names of each Managed Object in


your job creation request are syntactically correct.

2.

Check that all given counter names are legal.

Note
It is possible that a counter not yet activated may be dependent on one
or more Nokia counters that have been activated in a concurrent
measurement job. In this case, the counter that is dependent cannot be
activated.

8.3

Problems stopping PM measurement jobs


The following sections deal with troubleshooting tasks that you can carry
out to resolve the problem of stopping a measurement job.

8.3.1

Resolving problems with the stop function


To resolve PM IRP problems with creating measurement jobs, check the
following in the order listed below:
1.

Check PM licence availability

2.

Check the log file

3.

Check if the 3GPP PM IRP server is running and enable tracing

4.

Check the notification service

5.

Check the emugmamx processes

The troubleshooting tasks listed here are similar to those performed when
you encounter problems creating a PM measurement job. For detailed
instructions, see, section Resolving problems with the measurement job
creation function.

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

41 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

8.4

Problems listing PM measurement jobs


The troubleshooting tasks and their order of execution are similar to the
troubleshooting tasks for creating and stopping PM measurement jobs.
For more detailed instructions, see Resolving problems with the stop
function.
Note that it is possible that the PM measurement job was successfully
created and then deleted by the system. A measurement job is configured
for deletion according to this formula:
Example
Stop Time + Length of one Reporting Period + Configurable Delay

42 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Troubleshooting File Transfer IRP problems

Troubleshooting File Transfer IRP


problems
This chapter addresses possible problems with the File Transfer (FT) IRP.

9.1

Problems with FT IRP


The possible problems with the FT IRP are listed as follows:

9.1.1

The FTP IRP operation executes and returns unexpected error

The FTP IRP operation returns an empty list

Resolving File Transfer IRP problems


To resolve File Transfer IRP problems, check the following in the order
listed below:
1.

Check the log file: werlogA5A3.log

2.

Check the notification service

3.

Check for failed file transfers

For details, see the following instructions.

Check the log file


1.

Open the werlogA5A3.log file and check to see if any problems


have been logged in the file.

omc% $OMCLOGDIR/werlogA5A3.log

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

43 (45)

Troubleshooting 3GPP CORBA Northbound Interface Problems

2.

If there are problems in reported in the log file, correct them. If there
are no problems reported in the log file, continue with the next task,
Check the notification service.

Check the notification service


1.

Check if the notification adaptor server process, nuntiumx, is up and


running. To do this, enter the following at the command prompt:
omc% ps -ef grep -v grep grep
nuntiumx

Check the returned entries


1.

Check if too many entries are returned by the operation: list available
files. If there are too many entries, select a shorter time interval.

omc% ls -ltr $VARROT/uma/umanbi/pg_c3gcom/work/export wc


-

Check for failed file transfers


1.

Check for failed file transfers. To do this, enter the following at the
command prompt:

omc% ls -ltr $VARROOT/uma/umanbi/pg_c3gcom/work/c3gfit/failed wc


-l

2.

Check the number of files that the FT IRP subsystem currently


provides for download. To do this, enter the following at the
command prompt:

omc% ls -ltr $VARROOT/uma/umanbi/pg_c3gcom/work/export wc


-l

3.

Compare the number of successfully processed file transfers to the


failed file transfers. To determine the number of successful file
transfers, enter the following at the command prompt:

omc% ls -ltr $VARROOT/uma/umanbi/pg_c3gcom/work/c3gfit/ready wc


-l

44 (45)

# Nokia Siemens Networks

DN05171601
Issue 3-0 en

Where to find more information

10

Where to find more information


Related Nokia NetAct documentation
.

For general information on the 3GPP CORBA Northbound Interface,


see 3GPP CORBA Northbound Interface Principles.

For information on data flows, processes, and configuration files for


the 3GPP CORBA Northbound Interface, see 3GPP CORBA
Northbound Interface Technical Reference Guide.

All 3GPP specifications are available at http://www.3gpp.org.

DN05171601
Issue 3-0 en

# Nokia Siemens Networks

45 (45)