You are on page 1of 62

Mediations Troubleshooting

Mediations Troubleshooting

Contents
1 General Troubleshooting Tasks 3
1.1 Verifying and Correcting status of Mediations 4
1.1.1 General Procedure 4
1.1.2 NetAct Status Script 7
1.1.3 Mediation Framework Console 13
1.1.4 Managing Service Assemblies with MF Console 15
1.1.5 Mediations Logging and Tracing 21
2 Troubleshooting the Core Network 29
2.1 NE3S/WS Troubleshooting Cases 30
2.1.1 Network Element integration is not working properly 30
2.1.2 Alarm upload failure 32
2.1.3 PM Data not available 34
2.2 SNMP Troubleshooting Cases 35
2.2.1 Trap delivery is not working 35
2.2.2 Network element is not polled by Agent Supervision 38
2.2.3 SNMP AGENT NOT RESPONDING alarm is generated 38
2.3 XoH Troubleshooting Cases 39
2.3.1 Alarms are uploaded but NetAct Monitor shows a time-out error 39
2.3.2 XoH mediation reports alarms in Monitor 41
3 Troubleshooting the Radio Network 45
3.1 NWI3 Mediation 46
3.1.1 Integration data upload is not working 46
3.1.2 Alarms are not received 49
3.1.3 PM data is not received from OMS 51
3.2 Q3/MML Mediation 53

OS90538EN15GLA0
1
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

3.2.1 No alarm or event come from Q3 based NEs like BSC and SGSN 53
3.2.2 PM data is not received from BSC/SGSN 54
3.2.3 Generating upload file failed 56
3.2.4 Upload or file based plan provisioning ongoing 56
4 Exercises 57

OS90538EN15GLA0
2
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

1 General Troubleshooting Tasks

OS90538EN15GLA0
3
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

1.1 Verifying and Correcting status of Mediations
You may need to verify and correct the working status of the NetAct southbound
mediation in many error cases.
Follow the instructions below for checking the status of the following mediations:
 Q3 Mediations
 NWI3
 NE3S/WS
 XoH
 NX2S
 SNMP Northbound Interface
 FM ASCII Alarm Forwarding Northbound Interface
 3GGP CORBA NBI - Common Services
 3GPP CORBA PM NBI

1.1.1 General Procedure
1. Log in to any NetAct VM as the omc user.
2. Mediations run as services in the virtual machine, and they are managed by
netact_status.sh.
[omc]$ /opt/cpf/sbin/netact_status.sh status
The following is part of the command output:

[root@n85vm16 ~]# /opt/cpf/sbin/netact_status.sh status
Executing NetAct cluster status script
Status of NetAct
Node Service Status

n85vm03
ssh-n85vm03 started
dns-master started
netfs-n85vm03 started
dirsrv started
nslcd-n85vm03 started
omagentaf-n85vm03 started
ntp-n85vm03 started
cpfnx2s-n85vm03 started
tomcat_vcentplg started

OS90538EN15GLA0
4
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

3. Depending on the results of the netact_status.sh command, follow the first
or the secondchoice shown below:
 If the status of the service is started, then go to step 4.
Or
 If the status of the service is not started, then restart the service by
switching to the root user and entering:
[root]# smanager.pl start service <sm service name>

TIP
If the service is still not started, delete data folder and execute the below
commands:
• [root]# rm –rf /opt/oss/NSN-<sm service name>/smx/data/*
• [root]# smanager.pl start service <sm service name>

4. Even if the netact_status.sh command shows that the service is running, it
is still possible that a particular service assembly inside a service in the cluster is
not running. You can use the MF Console (Mediation Framework Console) to
check if a service assembly is in started state. Details about Mediation
Framework Console will be explained in next section.
5. If the status is Stopped, start the service assembly.
6. If the problem persists, send the mediation-specific log files
oss_activity*.log,oss_trace*.log, and oss_error*.log to the NSN
Technical Assistance Center. Note that you must collect the log files from each
VM node where the problematic service (services) is running.

OS90538EN15GLA0
5
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

General Troubleshooting Tasks
Correcting the status of a NetAct Mediation Service

You may need to verify and correct the working status of the NetAct southbound
mediation in many error cases. The main tasks are:

Verify If status is not If the status is If the status is
mediation “started” “started” but still “failed”
Service status restart the the service is remove the
• netact_status corresponding failing manage data
script service the status of directories of
• smanager.pl the SMX SAs. of the failing
script • Mediation mediation and
Framework restart the
console service
• Mediation
directories
• smanager.pl
script

1 © Nokia Solutions and Networks 2015

Fig. 1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OS90538EN15GLA0
. . . . .
6
. . . . . . . . . . . . . . . . . . . . . . . . . . . . © Nokia
. .Solutions
. .and.Networks
. . 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

1.1.2 NetAct Status Script
The NetAct Status script provides the status of the services running in the NetAct
cluster.

[root@n85vm16 ~]# /opt/cpf/sbin/netact_status.sh status
Executing NetAct cluster status script
Status of NetAct
Node Service Status
n85vm03
ssh-n85vm03 started
dns-master started
netfs-n85vm03 started
dirsrv started
nslcd-n85vm03 started
omagentaf-n85vm03 started
ntp-n85vm03 started
cpfnx2s-n85vm03 started
tomcat_vcentplg started
n85vm04
ssh-n85vm04 started
netfs-n85vm04 started
nslcd-n85vm04 started
db started
omagentaf-n85vm04 started
ntp-n85vm04 started
cpfnx2s-n85vm04 started
n85vm05
ssh-n85vm05 started
nfs started
dirsrv-secondary started
nslcd-n85vm05 started
omagentaf-n85vm05 started
ntp-n85vm05 started
cpfnx2s-n85vm05 started
n85vm06
ssh-n85vm06 started
netfs-n85vm06 started
nslcd-n85vm06 started
omagentaf-n85vm06 started
ntp-n85vm06 started
lb-unify-backup started
cpfnx2s-n85vm06 started
n85vm07
ssh-n85vm07 started
netfs-n85vm07 started
nslcd-n85vm07 started
omagent started
omagentaf-n85vm07 started
ntp-n85vm07 started
vcenterselfmon started
cpfnx2s-n85vm07 started
hpsim started
n85vm08
ssh-n85vm08 started
netfs-n85vm08 started
nslcd-n85vm08 started

OS90538EN15GLA0
7
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

ftp-n85vm08 started
omagentaf-n85vm08 started
ntp-n85vm08 started
cpfnx2s-n85vm08 started
pingserver-n85vm08 started
passwordchangews-n85vm08 started
tnameserver-n85vm08 started
nbi3gc started
nbiim started
rac3gp started
n85vm09
ssh-n85vm09 started
netfs-n85vm09 started
nslcd-n85vm09 started
ftp-n85vm09 started
omagentaf-n85vm09 started
ntp-n85vm09 started
cpfnx2s-n85vm09 started
pingserver-n85vm09 started
passwordchangews-n85vm09 started
tnameserver-n85vm09 started
nwi3 started
nwi3-http started
jacorb started
saucnt started
n85vm10
ssh-n85vm10 started
netfs-n85vm10 started
nslcd-n85vm10 started
ftp-n85vm10 started
omagentaf-n85vm10 started
ntp-n85vm10 started
osi started
cpfnx2s-n85vm10 started
pingserver-n85vm10 started
passwordchangews-n85vm10 started
tnameserver-n85vm10 started
q3user started
n85vm11
ssh-n85vm11 started
netfs-n85vm11 started
nslcd-n85vm11 started
omagentaf-n85vm11 started
ntp-n85vm11 started
lb-unify-primary started
cpfnx2s-n85vm11 started
socks started
n85vm12
ssh-n85vm12 started
netfs-n85vm12 started
nslcd-n85vm12 started
ftp-n85vm12 started
omagentaf-n85vm12 started
ntp-n85vm12 started
cpfnx2s-n85vm12 started
pingserver-n85vm12 started
passwordchangews-n85vm12 started
tnameserver-n85vm12 started
nx2s started
xoh started

OS90538EN15GLA0
8
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

ne3sws_dynamicadaptation started
generic_mediations started
n85vm13
ssh-n85vm13 started
netfs-n85vm13 started
nslcd-n85vm13 started
omagentaf-n85vm13 started
ntp-n85vm13 started
cpfnx2s-n85vm13 started
pingserver-n85vm13 started
passwordchangews-n85vm13 started
tnameserver-n85vm13 started
fmascii started
nbisnmp started
nbine3s started
n85vm14
ssh-n85vm14 started
netfs-n85vm14 started
nslcd-n85vm14 started
ftp-n85vm14 started
omagentaf-n85vm14 started
ntp-n85vm14 started
cpfnx2s-n85vm14 started
nbi3gcpm-n85vm14 started
pingserver-n85vm14 started
passwordchangews-n85vm14 started
tnameserver-n85vm14 started
n85vm15
ssh-n85vm15 started
netfs-n85vm15 started
nslcd-n85vm15 started
ftp-n85vm15 started
omagentaf-n85vm15 started
ntp-n85vm15 started
cpfnx2s-n85vm15 started
nbi3gcpm-n85vm15 started
pingserver-n85vm15 started
passwordchangews-n85vm15 started
tnameserver-n85vm15 started
n85vm16
ssh-n85vm16 started
dns-slave started
netfs-n85vm16 started
nslcd-n85vm16 started
omagentaf-n85vm16 started
ntp-n85vm16 started
cpfnx2s-n85vm16 started
common_mediations-n85vm16 started
httpd-n85vm16 started
pingserver-n85vm16 started
passwordchangews-n85vm16 started
tnameserver-n85vm16 started
n85vm17
ssh-n85vm17 started
netfs-n85vm17 started
nslcd-n85vm17 started
omagentaf-n85vm17 started
ntp-n85vm17 started
cpfnx2s-n85vm17 started
common_mediations-n85vm17 started

OS90538EN15GLA0
9
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

httpd-n85vm17 started
pingserver-n85vm17 started
passwordchangews-n85vm17 started
tnameserver-n85vm17 started
n85vm18
ssh-n85vm18 started
netfs-n85vm18 started
nslcd-n85vm18 started
omagentaf-n85vm18 started
ntp-n85vm18 started
cpfnx2s-n85vm18 started
fm_pipe-n85vm18 started
n85vm19
ssh-n85vm19 started
netfs-n85vm19 started
nslcd-n85vm19 started
omagentaf-n85vm19 started
ntp-n85vm19 started
cpfnx2s-n85vm19 started
fm_pipe-n85vm19 started
n85vm20
ssh-n85vm20 started
netfs-n85vm20 started
nslcd-n85vm20 started
ftp-n85vm20 started
omagentaf-n85vm20 started
ntp-n85vm20 started
was-n85vm20 started
dmgr-n85vm20 started
ihs-n85vm20 started
jetty-n85vm20 started
cpfnx2s-n85vm20 started
n85vm21
ssh-n85vm21 started
netfs-n85vm21 started
nslcd-n85vm21 started
ftp-n85vm21 started
omagentaf-n85vm21 started
ntp-n85vm21 started
was-n85vm21 started
ihs-n85vm21 started
jetty-n85vm21 started
cpfnx2s-n85vm21 started
n85vm22
ssh-n85vm22 started
netfs-n85vm22 started
nslcd-n85vm22 started
ftp-n85vm22 started
omagentaf-n85vm22 started
ntp-n85vm22 started
was-n85vm22 started
ihs-n85vm22 started
jetty-n85vm22 started
cpfnx2s-n85vm22 started
n85vm23
ssh-n85vm23 started
netfs-n85vm23 started
nslcd-n85vm23 started
ftp-n85vm23 started
omagentaf-n85vm23 started

OS90538EN15GLA0
10
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

ntp-n85vm23 started
was-n85vm23 started
ihs-n85vm23 started
jetty-n85vm23 started
cpfnx2s-n85vm23 started
n85vm26
nfs-n85vm26 started
system started
netfs-n85vm26 started
lb-n85vm26 started
db_crons started
n85vm27
netfs-n85vm27 started
lb-n85vm27 started
message started

n85vm28
netfs-n85vm28 started
base-n85vm28 started
cmessage started
httpd-n85vm28 started
osscore started
osscore2 started
nww started
mvi started
dns-namesurfer started

n85vm29
netfs-n85vm29 started
base-n85vm29 started
rep-n85vm29 started

n85vm30
netfs-n85vm30 started
sqm_eem-n85vm30 started
sqm_jboss-n85vm30 started
sqm_collector_npm started
dmgr_linas-n85vm30 started
nodeagent_linas-n85vm30 started
ihs_linas-n85vm30 started
was_linas-n85vm30 started

n85vm31
netfs-n85vm31 started
sqm_eem-n85vm31 started
sqm_jboss-n85vm31 started
nodeagent_linas-n85vm31 started
ihs_linas-n85vm31 started
was_linas-n85vm31 started

OS90538EN15GLA0
11
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

General Troubleshooting Tasks
NetAct cluster status

[root@n85vm16 ~]# /opt/cpf/sbin/netact_status.sh status
Executing NetAct cluster status script
Status of NetAct
Node Service Status
n85vm03
ssh-n85vm03 started
dns-master started
netfs-n85vm03 started
dirsrv started
nslcd-n85vm03 started
omagentaf-n85vm03 started
ntp-n85vm03 started
cpfnx2s-n85vm03 started
tomcat_vcentplg started
n85vm04
ssh-n85vm04 started
netfs-n85vm04 started
nslcd-n85vm04 started
db started
omagentaf-n85vm04 started
ntp-n85vm04 started
cpfnx2s-n85vm04 started
1 © Nokia Solutions and Networks 2015

Fig. 2

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OS90538EN15GLA0
. . . . .
12
. . . . . . . . . . . . . . . . . . . . . . . . . . . . © Nokia
. .Solutions
. .and.Networks
. . 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

1.1.3 Mediation Framework Console
MF Console is a application intended to monitor, troubleshoot and administer MF
runtime (Apache Servicemix instance).
The purpose of the MF Console is to get a consolidated view of the runtime
information to troubleshoot the system. It is integrated with tracer to trace the logs for
particular use-cases. It monitors the performance of the different components in the
servicemix runtime.
The Scope of MF console is to monitor and troubleshoot the apache servicemix
instance that includes:
 interfacing to start/stop existing mediations
 viewing mediation information
 viewing mediations running on the given MF instance
 viewing different components within an mediation

Usage
The MFC.sh script will be available under the mf-console folder within the servicemix
folder.
Use the command
#./MF.sh <command> [args]
to use particular operations.

Commands

Get Displays all available static information for the Servicemix
instance (list of Service Assemblies, list of Service Units and list
of End Points)
dispSA Retrieves the list of service assemblies deployed in the
Servicemix instance along with their status
dispMI Retrieves the mediation information (list of service units and list
of endpoints) for the given service assembly
start Starts the service assembly
stop Stops the service assembly
shutDown Shuts down the service assembly

OS90538EN15GLA0
13
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

General Troubleshooting Tasks
Mediation Framework Console
MF Console is an application to monitor, troubleshoot and administer MF runtime
(Apache Servicemix instance). The MFC.sh script is available under the mf-
console folder within the servicemix instance folder located in the VM running the
corresponding mediation. Use the command
#./MF.sh <command> [args]

Command Description
Displays all available static information for the Servicemix instance
Get
(list of Service Assemblies, list of Service Units and list of End Points)
Retrieves the list of service assemblies deployed in the Servicemix
dispSA
instance along with their status
Retrieves the mediation information (list of service units and list of
dispMI
endpoints) for the given service assembly
start Starts the service assembly
stop Stops the service assembly
shutDown Shuts down the service assembly
1 © Nokia Solutions and Networks 2015

Fig. 3

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OS90538EN15GLA0
. . . . .
14
. . . . . . . . . . . . . . . . . . . . . . . . . . . . © Nokia
. .Solutions
. .and.Networks
. . 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

1.1.4 Managing Service Assemblies with MF Console
To see the correct status of the components of a specific Mediation, execute the
commands below:
1. Log on as root user in the VM node that is currently running the mediation
(NetAct service) in question
2. To get a list of the service assemblies deployed in one specific Mediation, enter
the following commands:
[root]# cd /opt/oss/<Mediation Folder>/smx/mf-console
[root]# ./MFC.sh dispSA
The Mediation folders are as follows:
 Q3 Mediation : /opt/oss/NSN-q3_mediations/smx/mf-console
 NWI3: /opt/oss/NSN-nwi3/smx/mf-console
 NE3S/WS: /opt/oss/NSN-common_mediations/smx/mf-console
 XoH: /opt/oss/NSN-xoh/smx/mf-console
 NX2S: /opt/oss/NSN-nx2s/smx/mf-console
 SNMP Northbound Interface: /opt/oss/NSN-nbisnmp/smx/mf-console
 FM ASCII NBI: /opt/oss/NSN-fmascii/smx/mf-console
 3GGP CORBA NBI - Common: /opt/oss/NSN-nbi3gc/smx/mf-
console
 3GPP CORBA PM NBI: /opt/oss/NSN-nbi3gcpm/smx/mf-console

The output of the MFConsole commands indicates the status of each service
assembly of the service in question in the following way:
INFO - DispSACommand - <service_assembly> -- Started

3. If the status of the service assembly is "stopped" or "shutdown" start the service
assembly by entering:
[root]# ./MFC.sh start -sa <service_assembly

OS90538EN15GLA0
15
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Examples of Service Assemblies in different Mediations:

NWI3
[root@n85vm09 mf-console]# cd /opt/oss/NSN-nwi3/smx/mf-console
[root@n85vm09 mf-console]# ./MFC.sh dispSA
INFO - DispSACommand - Service Assemblies available are:
INFO - DispSACommand - nwi3-mediation-pm-file-transfer-sa -- Started
INFO - DispSACommand - nwi3-mediation-sa -- Started
INFO - DispSACommand - nas-nw3_cm-jbi-sa -- Started
INFO - DispSACommand - nwi3-mediation-logUpload-sa -- Started
INFO - DispSACommand - nwi3-mediation-audit-trail-sa -- Started
INFO - DispSACommand - nwi3-lic-mediation-sa -- Started
INFO - DispSACommand - nwi3-mediation-fm-sa -- Started
INFO - DispSACommand - nwi3-mediation-um-sa -- Started
INFO - DispSACommand - nwi3-mediation-certificate-management-sa --
Started
INFO - DispSACommand - nwi3-mediation-pm-file-query-sa -- Started

Q3/MML
[root@n85vm10 mf-console]# cd /opt/oss/NSN-q3_mediations/smx/mf-console
[root@n85vm10 mf-console]# ./MFC.sh dispSA
INFO - DispSACommand - Service Assemblies available are:
INFO - DispSACommand - q3nebr-sa -- Started
INFO - DispSACommand - q3hwm-sa -- Started
INFO - DispSACommand - q3stack-sa -- Started
INFO - DispSACommand - q3slt-sa -- Started
INFO - DispSACommand - q3common-sa -- Started
INFO - DispSACommand - q3neum-sa -- Started
INFO - DispSACommand - sa-SmixHeartbeat -- Started
INFO - DispSACommand - q3pm-sa -- Started
INFO - DispSACommand - q3aom-sa -- Started
INFO - DispSACommand - q3swm-sa -- Started
INFO - DispSACommand - q3tm-sa -- Started
INFO - DispSACommand - q3fm-sa -- Started
INFO - DispSACommand - q3lm-sa -- Started
INFO - DispSACommand - q3atl-sa -- Started

OS90538EN15GLA0
16
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

XoH
[root@n85vm12 mf-console]# cd /opt/oss/NSN-xoh/smx/mf-console
[root@n85vm12 mf-console]# ./MFC.sh dispSA
INFO - DispSACommand - Service Assemblies available are:
INFO - DispSACommand - xohslt-sa -- Started
INFO - DispSACommand - xohpm-sa -- Started
INFO - DispSACommand - xohfm-sa -- Started
INFO - DispSACommand - common-sa -- Started

Common Mediations: NE3S/WS – 3GPP Corba Southbound – NE3S/SNMP
Mediation
[root@n85vm16 mf-console]# cd /opt/oss/NSN-common_mediations/smx/mf-console
[root@n85vm16 mf-console]# ./MFC.sh dispSA
INFO - DispSACommand - Service Assemblies available are:
INFO - DispSACommand - mf-3gpp-corba-smi-bc-sa -- Started
INFO - DispSACommand - m3gc-sbm-seq-push-consumer-sa -- Started
INFO - DispSACommand - com.nsn.cms9000.active -- Started
INFO - DispSACommand - atlc-mf-sa -- Started
INFO - DispSACommand - m3gcsbm-fm-adapter-oes-sa -- Started
INFO - DispSACommand - com.nsn.juniper.base -- Started
INFO - DispSACommand - mf-3gpp-corba-sbm-jbi4corba-v3-sa -- Started
INFO - DispSACommand - ne3sws-manager-oes-sa -- Started
INFO - DispSACommand - ne3s-snmp-manager-sa -- Started
INFO - DispSACommand - com.nsn.pm.ftp.mediation-sa -- Started
INFO - DispSACommand - m3gcsbm-cs-adapter-oes-sa -- Started
INFO - DispSACommand - m3gcsbm-bcm-adapter-oes-sa -- Started
INFO - DispSACommand - m3gc-sbm-fm-jbi4corba-v3-sa -- Started
INFO - DispSACommand - mf-3gpp-corba-sbm-cm-sa -- Started
INFO - DispSACommand - ne3sws-manager-https-sa -- Started
INFO - DispSACommand - m3gcsbm-pm-adapter-oes-sa -- Started
INFO - DispSACommand - m3gc-sbm-fm-jbi4corba-sa -- Started
INFO - DispSACommand - m3gcs-cs-jbi4corba-sa -- Started
INFO - DispSACommand - sa-SmixHeartbeat -- Started
INFO - DispSACommand - com.nsn.fns -- Started
INFO - DispSACommand - com.nsn.hp.hphw -- Started
INFO - DispSACommand - mf-3gpp-corba-sbm-jbi4corba-sa -- Started
INFO - DispSACommand - mf-3gpp-corba-mediations-common-sa -- Started
INFO - DispSACommand - mf-corba-notification-sa -- Started

OS90538EN15GLA0
17
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

INFO - DispSACommand - m3gcsbm-fpe-adapter-oes-sa -- Started
INFO - DispSACommand - mf-3gpp-corba-sbm-fm-sa -- Started
INFO - DispSACommand - bss-ivm-med-sa -- Started
INFO - DispSACommand - ne3sws-manager-sa -- Started
[root@n85vm16 mf-console]#

SNMP Northbound Interface
[root@n85vm08 mf-console]# cd /opt/oss/NSN-nbisnmp/smx/mf-console
[root@n85vm13 mf-console]# ./MFC.sh dispSA
INFO - DispSACommand - Service Assemblies available are:
INFO - DispSACommand - AlarmIRP-Operation-SA -- Started
INFO - DispSACommand - nbisnmp-agent-sa -- Started
INFO - DispSACommand - nbitrap-sa -- Started
INFO - DispSACommand - nbisnmp-access-sa -- Started

3GPP Corba NBI Common Services (nbi3gc service):
[root@n85vm08 mf-console]# cd /opt/oss/NSN-nbi3gc/smx/mf-console
[root@n85vm08 mf-console]# ./MFC.sh dispSA
INFO - DispSACommand - Service Assemblies available are:
INFO - DispSACommand - AlarmIRP-Notification-SA --
Started
INFO - DispSACommand - Common-SA -- Started
INFO - DispSACommand - FTIRP-Notification-SA -- Started
INFO - DispSACommand - NotificationIRP-SA -- Started
INFO - DispSACommand - Epirp-SA -- Started
INFO - DispSACommand - BasicCMAccessIRP-SA -- Started
INFO - DispSACommand - AlarmIRP-Operation-SA -- Started
INFO - DispSACommand - Csirp-SA -- Started
INFO - DispSACommand - KernelCmIRP-SA -- Started
INFO - DispSACommand - FTIRP-Operation-SA -- Started
[root@n85vm08 mf-console]#

OS90538EN15GLA0
18
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

General Troubleshooting Tasks
Mediation Framework Console: Service Assemblies

[root@n85vm09 mf-console]# cd /opt/oss/NSN-nwi3/smx/mf-console
[root@n85vm09 mf-console]# ./MFC.sh dispSA
INFO - DispSACommand - Service Assemblies available are:
INFO - DispSACom - nwi3-mediation-pm-file-transfer-sa -- Started
INFO - DispSACommand - nwi3-mediation-sa -- Started
INFO - DispSACommand - nas-nw3_cm-jbi-sa -- Started
INFO - DispSACommand - nwi3-mediation-logUpload-sa -- Started
INFO - DispSACommand - nwi3-mediation-audit-trail-sa -- Started
INFO - DispSACommand - nwi3-lic-mediation-sa -- Started
INFO - DispSACommand - nwi3-mediation-fm-sa -- Started
INFO - DispSACommand - nwi3-mediation-um-sa -- Started
INFO - DispSACommand - nwi3-mediation-certificate-management-sa --
Started
INFO - DispSACommand - nwi3-mediation-pm-file-query-sa -- Started

1 © Nokia Solutions and Networks 2015

Fig. 4

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . .
OS90538EN15GLA0 . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
. Solutions
© Nokia . . and. Networks
. . .2015. . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

General Troubleshooting Tasks
Mediation Framework Console: Starting / stopping Service Assemblies

[root@n85vm09 mf-console]# ./MFC.sh dispSA
INFO - DispSACommand - Service Assemblies available are:
INFO - DispSACommand - nwi3-mediation-pm-file-transfer-sa -- Started
INFO - DispSACommand - nwi3-mediation-sa -- Started
INFO - DispSACommand - nas-nw3_cm-jbi-sa -- Started
INFO - DispSACommand - nwi3-mediation-logUpload-sa -- Stopped
INFO - DispSACommand - nwi3-mediation-audit-trail-sa -- Started
INFO - DispSACommand - nwi3-lic-mediation-sa -- Started
INFO - DispSACommand - nwi3-mediation-fm-sa -- Started
INFO - DispSACommand - nwi3-mediation-um-sa -- Started
INFO - DispSACommand - nwi3-mediation-certificate-management-sa -- Started
INFO - DispSACommand - nwi3-mediation-pm-file-query-sa -- Started

[root@n85vm09 mf-console]#
[root@n85vm09 mf-console]# ./MFC.sh start -sa nwi3-mediation-
logUpload-sa
INFO - StartCommand - nwi3-mediation-logUpload-sa service
Assembly started successfully
[root@n85vm09 mf-console]#
1 © Nokia Solutions and Networks 2015

Fig. 5

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
OS90538EN15GLA0
20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
© Nokia Solutions and Networks 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

1.1.5 Mediations Logging and Tracing
Logging and tracing is the process of automatically recording information about the
behavior of the system, components and individual workflows to assist in:
• Debugging
• Troubleshooting
• Auditing activities
The following table lists the differences between logging and tracing:

Logging` Tracing

Used to make a note of important events Used to record any information that helps
occurring in the system. in debugging problems with the code.
Used to capture state changes, to Used to indicate which methods are
indicate warnings, or to indicate errors. called, what data is passed into a
method, what values are returned out of
a method, and what data is returned from
calls to methods beyond the boundary of
a component.
Log messages are easy to understand. Trace messages are detailed and
technical in nature, so they are of use
mainly to the developer community who
wrote the application. With tracing turned
on, it should be possible to understand
most of the problems that occur in the
system.
By default, logging is turned on and By default, tracing is turned off. The
consumes less space. messages are of high volume and
consume system resources.

OS90538EN15GLA0
21
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

General Troubleshooting Tasks
Mediations Logging and Tracing
Logging Tracing

Used to make a note of important events Used to record any information that helps in
occurring in the system. debugging problems with the code.

Used to indicate which methods are called,
what data is passed into a method, what values
Used to capture state changes, to
are returned out of a method, and what data is
indicate warnings, or to indicate errors.
returned from calls to methods beyond the
boundary of a component.

Trace messages are detailed and technical in
nature, so they are of use mainly to the
developer community who wrote the
Log messages are easy to understand.
application. With tracing turned on, it should be
possible to understand most of the problems
that occur in the system.
By default, tracing is turned off. The messages
By default, logging is turned on and
are of high volume and consume system
consumes less space.
resources.
1 © Nokia Solutions and Networks 2015

Fig. 6

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OS90538EN15GLA0
. . . . .
22
. . . . . . . . . . . . . . . . . . . . . . . . . . . . © Nokia
. .Solutions
. .and.Networks
. . 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

Log files provide information about events or errors in the system. They are
automatically created and maintained by the system. The administrator uses the log
files to monitor and debug problems.
NetAct system requirements defines the content that each component must provide
into logs and traces for different scopes under different circumstances.
The following table lists the NetAct system requirements for logging and tracing,
highlighting the contents of log/trace. It also shows the log or trace types provided by
NetAct system logging service. This requirement applies to all applications and
mediations

Logging Type Description Log Name Log Level
Event Logging It shows the activity in the Activity Log INFO
component at a broader level.
Service Mix Startup and shutdown of the Servicemix-Log INFO
Event Logging component.
Event Tracing Start and stop of the workflow Trace Log CONFIG
within the component, including FINE
parameters.
FINER
Request to another component,
FINEST
including parameters and results.
Request from another component,
including parameters.
Component is not able to recover
from an error situation.
Error Logging Functional problems that are Error Log SEVERE
caused by users, run-time
FATAL
environment or faults in the
component. ERROR
Component is not able to recover WARNING
from an error situation. WARN
CONFIG

OS90538EN15GLA0
23
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

General Troubleshooting Tasks
Mediations Logging and Tracing

Logging Type Description Log Name Log Level
It shows the activity in the component at a
Event Logging Activity Log INFO
broader level.
Activity log in
Service Mix Event
Startup and shutdown of the component. Servicemix-Log INFO
Logging
directory
• Start and stop of the workflow within the
component, including parameters.
CONFIG
• Request to another component, including
parameters and results. Request from FINE
Event Tracing Trace Log
another component, including FINER
parameters.
FINEST
• Component is not able to recover from
an error situation.
SEVERE
• Functional problems that are caused by FATAL
users, run-time environment or faults in ERROR
Error Logging the component. Error Log
• Component is not able to recover from WARNING
an error situation. WARN
1 © Nokia Solutions and Networks 2014
CONFIG

Fig. 7

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OS90538EN15GLA0
. . . . .
24
. . . . . . . . . . . . . . . . . . . . . . . . . . . . © Nokia
. .Solutions
. . and.Networks
. . 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

Each mediation component defines the default logging. The common guideline is that
the basic activity and error logs are named same way.

For example, in the case of XoH-mediation the logging directory is
/var/opt/oss/log/xoh

[root@n85vm12 xoh]# ll
total 24248
drwxr-x--- 2 xohuser sysop 4096 Sep 15 12:35 NSN-mf_component
-rw-r--r-- 1 xohuser sysop 3437559 Oct 5 03:15 oss_activity0_0.log
-rw-r--r-- 1 xohuser sysop 10485818 Sep 28 16:20 oss_activity0_1.log
-rw-r--r-- 1 xohuser sysop 10485863 Sep 5 03:54 oss_activity0_2.log
-rw-r--r-- 1 xohuser sysop 380392 Oct 3 10:21 oss_error0_0.log
-rw-r--r-- 1 xohuser sysop 0 Aug 16 10:33 oss_trace0_0.log
drwx------ 2 xohuser sysop 4096 Aug 16 10:33 Performance-Log
drwx------ 2 xohuser sysop 4096 Aug 16 10:33 Servicemix-Log
[root@n85vm12 xoh]#

Mediation Log Directories Trace Script

NWI3 SBI /var/opt/oss/log/ nwi3 /opt/oss/NSN-
nwi3/bin/nwi3_setLoggingLevel.sh <level>
change switches logging between
PRODUCTION and TROUBLESHOOT

Q3/MML SBI /var/opt/oss/log/q3user /opt/oss/NSN-q3med-
/var/opt/oss/log/q3mml common/bin/set_q3user_logger_level.sh –
fragment LM –level <level>
Available values for command line parameter –
fragment: NEBR , LM , SLT , ATL , UM , PM , FM
, SWM , AoM ,HWM/
Available values for parameter – level: FATAL ,
ERROR , WARN , INFO , DEBUG , TRACE ,
OFF ,ALL

NE3S/WS /var/opt/oss/log/ Manually modify the file: /opt/oss/NSN-
SBI common_mediations common_mediations/smx/conf/log4j.xml:
Change the threshold value from OFF to
DEBUG for mf-trace parameter.
Change the root level of the log from INFO
to DEBUG

OS90538EN15GLA0
25
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

XoH SBI /var/opt/oss/log/ xoh #/opt/oss/NSN-
jbi_swp/bin/set_logger_level.sh -instance
xoh -logger com.nsn.oss.xoh -level <level>
The possible values for level are INFO
(default), TRACE, DEBUG, WARN, ERROR
and FATAL

NX2S SBI /var/opt/oss/log/ nx2s #/opt/oss/NSN-jbi_swp/bin/
set_logger_level.sh –instance nx2s –logger
com.nsn.oss.nx2s –level <level>
The log levels include FATAL, ERROR,
WARN, INFO, DEBUG and TRACE

FMASCII /var/opt/oss/log/ fmascii # /etc/init.d/fmascii log <level>
NBI The possible values of <level> are ERROR,
INFO and DEBUG.

SNMP NBI /var/opt/oss/log/ # nbisnmp log <level>
nbisnmp The possible values of <level> are INFO,
DEBUG and TRACE.

3GPP Corba /var/opt/oss/log/ nbigc # smx-clt log <level>
Common The possible values of <level> are INFO,
NBI DEBUG and TRACE.

3GPP Corba /var/opt/oss/log/ # nbi3gcpm log <level>
PM NBI nbi3gcpm The possible values of <level> are INFO,
DEBUG and TRACE.

3GPP Corba  /var/opt/oss/log/ Change the log level by changing the value
Bulk CM NBI rac/rac3gp in of <level value> in rac3gp_log4j.xml file in
rac3gp node rac3gp node:
The possible values of <level value> are
 /var/opt/oss/log/
INFO, DEBUG, WARN, ERROR and FATAL.
rac in WAS
node Modify the trace level in WAS node as
"omc"user:
[omc]$ /opt/nokia/oss/bin/set_trace_level.sh
-logger rac.rrcsvc -level <levelName> –
create
The possible values of <levelName>, from
higher level to lower level, are SEVERE,
WARNING, INFO, CONFIG, FINE , FINER
and FINEST. And there are two special
values for <level value>, which are ALL and
OFF. ALL means all the messages are
logged and OFF means no message is
logged

OS90538EN15GLA0
26
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

General Troubleshooting Tasks
Mediations Logging and Tracing (1/2)
Mediation Log Directories Trace Script / Configuration File
#/opt/oss/NSN-nwi3/bin/nwi3_setLoggingLevel.sh <level>
NWI3 SBI /var/opt/oss/log/ nwi3
<level> can be PRODUCTION or TROUBLESHOOT

# /opt/oss/NSN-q3med-common/bin/set_q3user_logger_level.sh –
fragment LM –level DEBUG
/var/opt/oss/log/q3user
Q3/MML SBI Available values for fragment: NEBR , LM , SLT , ATL , UM , PM ,
/var/opt/oss/log/q3mml FM, SWM , AoM ,HWM
Available values for command line parameter – level: FATAL , ERROR,
WARN , INFO , DEBUG , TRACE , OFF ,ALL

Manually modify the file: /opt/oss/NSN-
common_mediations/smx/conf/log4j.xml:
/var/opt/oss/log/
NE3S/WS SBI Change the threshold value from OFF to DEBUG for mf-trace
common_mediations
parameter.
Change the root level of the log from INFO to DEBUG
#/opt/oss/NSN-jbi_swp/bin/
set_logger_level.sh -instance xoh -logger com.nsn.oss.xoh -level
XoH SBI /var/opt/oss/log/ xoh <level>
The possible values for level are INFO (default), TRACE, DEBUG,
WARN, ERROR and FATAL
# /opt/oss/NSN-jbi_swp/bin/
set_logger_level.sh –instance nx2s –logger com.nsn.oss.nx2s –
NX2S SBI /var/opt/oss/log/ nx2s level DEBUG
1 © Nokia Solutions and Networks 2014 The log levels include FATAL, ERROR, WARN, INFO, DEBUG and
TRACE

Fig. 8

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . .
OS90538EN15GLA0 . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
. Solutions
© Nokia . . and. Networks
. . .2015. . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

General Troubleshooting Tasks
Mediations Logging and Tracing (2/2)

Mediation Log Directories Trace Script / Configuration File

FMASCII NBI /var/opt/oss/log/ fmascii # /etc/init.d/fmascii log <level>
The possible values of <level> are ERROR, INFO and DEBUG.
SNMP NBI /var/opt/oss/log/ # nbisnmp log <level>
nbisnmp The possible values of <level> are INFO, DEBUG and TRACE.
3GPP Corba /var/opt/oss/log/ nbigc #smx-clt log <level>
Common NBI The possible values of <level> are INFO, DEBUG and TRACE.
3GPP Corba /var/opt/oss/log/ # nbi3gcpm log <level>
PM NBI nbi3gcpm The possible values of <level> are INFO, DEBUG and TRACE.
3GPP Corba • /var/opt/oss/log/rac/ • Change the log level by changing the value of <level value> in
Bulk CM NBI rac3gp in rac3gp node rac3gp_log4j.xml file in rac3gp node:
The possible values of <level value> are INFO, DEBUG, WARN,
• /var/opt/oss/log/rac in ERROR and FATAL.
WAS node • Modify the trace level in WAS node as "omc"user:
[omc]$ /opt/nokia/oss/bin/set_trace_level.sh -logger rac.rrcsvc -
level <levelName> –create
The possible values of <levelName>, from higher level to lower level,
are SEVERE, WARNING, INFO, CONFIG, FINE , FINER and
FINEST. And there are two special values for <level value>, which are
ALL and OFF. ALL means all the messages are logged and OFF
means no message is logged
1 © Nokia Solutions and Networks 2015

Fig. 9

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OS90538EN15GLA0
. . . . .
28
. . . . . . . . . . . . . . . . . . . . . . . . . . . . © Nokia
. .Solutions
. .and.Networks
. . 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

2 Troubleshooting the Core Network

OS90538EN15GLA0
29
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

2.1 NE3S/WS Troubleshooting Cases
2.1.1 Network Element integration is not working properly
The possible cause may be the agent registration data is lost or corrupted in the
mediation while performing operations such as upgrade.

Solution
Re-register the network element:
1. Execute the following SQL query by providing GID of the agent managed object:
Delete from COMMON_MEDIATIONS.JBOSS_CACHE_DATA where
FQN='/jboss/NE3SWSMOFAgentInfo-"GID_OF_AGENT_OBJECT"';
Registration entries of the agent in the mediation are cleared.
2. Perform upload operation which registers the agent in the mediation.

OS90538EN15GLA0
30
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Troubleshooting Core Network
NE3S/WS Mediation To check connectivity issues
ping the NE3S/WS agent:
Network Element integration is not working properly # wget http:://<agent
ip/hostname>:port
The possible cause may be the agent registration data is
lost or corrupted in the mediation while performing operations such as upgrade.

Solution: Re-register the network element:

• Execute the following SQL query by providing GID of the agent managed object:

Delete from COMMON_MEDIATIONS.JBOSS_CACHE_DATA where
FQN='/jboss/NE3SWSMOFAgentInfo-"GID_OF_AGENT_OBJECT"';
Registration entries of the agent in the mediation are cleared.

• Perform integration data upload operation which registers the agent in the
mediation.

Please review the chapter document for more NE3S/WS Troubleshooting cases
1 © Nokia Solutions and Networks 2015

Fig. 10

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . .
OS90538EN15GLA0 . . . . . . . . . . . . . . . . . . . . . . . . . . .
31
. Solutions
© Nokia . . and. Networks
. . .2015. . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

2.1.2 Alarm upload failure
Possible Causes
1. Manager is not registered and the com.nsn.oss.mf.mediation.ne3sws
service is not available.
Solution
Check the Service Assembly (SA) deployment:
a) Log in to the common mediations VM and check the status of Service
Assemblies via MF Console
b) Correct the status of stopped service Assemblies
c) If problem continues, service asemblies status can also be verified by
navigating to ./opt/oss/NSN-
common_mediations/smx/data/smx/serviceassemblies
Check all the NE3SWS related SA's deployment in the respective folders, the
value of the state.xml file must be started. Else, restart the affected
service assembly or restart the whole common_mediation service through
smanager.pl

2. Registration failure and 9047 Alarm is raised due to registration failure.
Solution
Check if the request has been sent to agent:
a) See Mediation logs in the common mediations VM to check if upload was
triggered.
Navigate to /var/opt/oss/log/common_mediations, and in the
oss_activity*.log and oss_trace*.log files, search for the Object
DN, for example, PLMN-XX/NTHLRFE-XX.
b) Check JOMA logs if the request has been acknowledged. Navigate to
.../R_ESYMAC34_8.0_snmp4j/extras/esymacstarter/logEvents.l
og.0.
c) Check the Instructions and Supplementary information field of the Alarm to
check why 9047 alarm is raised.
d) Verify that the HTTP port provided while creating NE3SWS Interface is correct.
e) Verify the NEAC credentials.
f) Verify that the NE3SWS interface object instance is 1 (for example, NE3SWS-
1).
g) Verify the Host name that is provided in MO properties of the Monitor.

OS90538EN15GLA0
32
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

3. Check lock manager issues:
Navigate to /var/opt/oss/log/common_mediations/ and check the
oss_error0_0.log file.
4. Request has not reached the agent.
Solution
a) Check connectivity issue:
Execute the following command to ping agent:
wget http:://<agent ip/hostname>:port
If the connection is not successful, contact Network Administrator.
b) Check the LB configuration and Notification URL in the mf-
ne3soap.properties file present in /opt/oss/NSN-
common_mediations/smx/mf-conf in common mediations node.
5. OFaS response has not reached mediations:
Solution
To check if OFaS response has not reached mediations:
a) Check JOMA logs to verify if request has been sent wrong notification URL.
Navigate to
.../R_ESYMAC34_8.0_snmp4j/extras/esymacstarter/logEvents.l
og.0.
b) Check if JOMA response is forwarded to NE3SWS mediation consumer URL
c) Check firewall to verify connectivity problems between agent and notification
URL in NetAct mediation.

OS90538EN15GLA0
33
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

2.1.3 PM Data not available
The PM data does not reach the NetAct Performance Management database or
reporting tools or both.
Possible causes
1. Registration failure and 9047 Alarm is raised due to registration failure.
Solution
Check if the request has been sent to agent:
a) See Mediation logs in the common mediations VM to check if upload was
triggered.
Navigate to /var/opt/oss/log/common_mediations, and in the
oss_activity*.log and oss_trace*.log files, search for the Object
DN, for example, PLMN-XX/NTHLRFE-XX.
b) Check JOMA logs if the request has been acknowledged. Navigate to
.../R_ESYMAC34_8.0_snmp4j/extras/esymacstarter/logEvents.log.0.
c) Check the Instructions and Supplementary information field of the Alarm to
check why 9047 alarm is raised.
d) Verify that the HTTP port provided while creating NE3SWS Interface is correct.
e) Verify the NEAC credentials.
f) Verify that the NE3SWS interface object instance is 1 (for example, NE3SWS-
1).
g) Verify the Host name that is provided in MO properties of the Monitor
2. Check lock manager issues: Navigate to
/var/opt/oss/log/common_mediations/ and check
theoss_error0_0.log file
3. The request file is not being picked up by file poller.
Check if file poller is configured for the wrong directory:
Navigate to /opt/oss/NSN-common_mediations/smx/mf-conf and find
the response and request directory in the mf-ne3soap-oes.properties
4. Request has not reached agent.
Check connectivity issue:
a) Execute the following command to ping Agent: wget http:://<agent
ip/hostname>:port
b) Check the LB configuration and Notification URL in the mf-
ne3soap.properties file in /opt/oss/NSN-
common_mediations/smx/mf-conf in common mediations node.

OS90538EN15GLA0
34
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

2.2 SNMP Troubleshooting Cases
2.2.1 Trap delivery is not working
Problem
When trap delivery is not working, the alarms are not present in FM DB.
The mediation receives trap from load balancer port 262 and load balancer port 162.
Possible causes
 Traps are not received
 Traps are received, but the mediation is not listening to the traps
 The mediation is up and running, and is listening to the traps, but the traps
are not auto-forwarded to the correct port

Solution
1. Check that traps are received
a) Log in to any NetAct VM, and switch to root user.
b) To see if the traps are received, execute tcpdump command.
#tcpdump -i eth0 'udp port 262'
When traps are received, tcpdump command displays the following:
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode listening on eth0, link-type EN10MB (Ethernet), capture size
96 bytes 10:33:11.011232 IP karadhcp18947.ntc.nokia.com.2564
>urho.ntc.nokia.com.snmptrap: Trap(35) E:nokia.7 1.1.1.1
enterpriseSpecifics=1 0 .iso.org.dod.internet.private=[|snmp]

2. Check that the mediation is listening to traps
a) Log in to any NetAct VM, and switch to root user.
b) Check that the mediation is up and running and listening to traps.
#netstat -ulp | grep 16262
When the mediation is up and running and listening to traps, the following
message appears:
udp 0 0 *:16262 *:*3421/java

OS90538EN15GLA0
35
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

3. Check trap auto-forwarding
a) Log in to any NetAct VM, and switch to root user.
b) Check that the traps are auto-forwarded from port 262 and load balancer port
162 to port 16262.
#service iptables status
When the autoforwarding works, the following message appears:
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 REDIRECT udp -- 0.0.0.0/0 0.0.0.0/0 udp
dpt:262 redir ports 16262
2 REDIRECT udp -- 0.0.0.0/0 0.0.0.0/0 udp
dpt:162 redir ports 16262

4. Check trap by mediation
a) Open SNMP mediation logs. The location is:
/var/opt/oss/log/snmp-mus/oss_trace0_0.log
The log is available on WAS VM node.
b) Ensure that the log contains a list of message bindings and values.

OS90538EN15GLA0
36
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Troubleshooting Core Network
SNMP Mediation

Trap delivery is not working
When trap delivery is not working, the alarms are not present in FM DB. The mediation receives trap from
load balancer port 262 and load balancer port 162.
Possible causes
• Traps are not received, traps are received, but the mediation is not listening to the traps OR the
mediation is up and running, and is listening to the traps, but the traps are not auto -forwarded to the
correct port.

Check that traps are received
1. Log in to any NetAct VM, and switch to root user.
2. To see if the traps are received, execute tcpdump command.
# tcpdump -i eth0 'udp port 262'
When traps are received, tcpdump command displays the following:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
10:33:11.011232 IP karadhcp18947.ntc.nokia.com.2564
>urho.ntc.nokia.com.snmptrap: Trap(35) E:nokia.7 1.1.1.1
enterpriseSpecifics=1 0 .iso.org.dod.internet.private=[|snmp]

Please review the chapter document for more SNMP Troubleshooting cases
1 © Nokia Solutions and Networks 2015

Fig. 11

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . .
OS90538EN15GLA0 . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
. Solutions
© Nokia . . and. Networks
. . .2015. . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

2.2.2 Network element is not polled by Agent Supervision
The network element is not polled by agent supervision.
Agent supervision is enabled in the adaptation fragment, however the network
element is not monitored.
Solution
Perform the following to enable agent supervision on the network element:
1. In the SNMP FM Mediator Application Select Agent Supervision Configuration.
2. Select the Agent Polling Active check box.
3. Click Update.
SNMP Mediation Agent Polling Configuration has been updated
message appears and the configuration is successfully updated.

2.2.3 SNMP AGENT NOT RESPONDING alarm is generated
Although SNMP agent is up at the managed element, the related alarm is generated
because of a network routing problem, wrong parameter configuration, or firewall
blocking. SNMP AGENT NOT RESPONDING alarm is generated, but the SNMP
agent is up and running on the related host.

Solution
1. Check IP routing configuration and Firewall configuration
The firewall must allow SNMP (UDP) traffic from the virtual machine to the
managed element (SNMPGET) and from the managed element to the virtual
machine (SNMP trap). The firewall must allow ICMP (ping) traffic from the virtual
machine to the managed element as well.
2. Check SNMP Read Access configuration.
In Network Element Access Control, check the SNMP Read Access service
type of corresponding object. The SNMP Read Access must be the same as
informed by the managed element administrator.
3. Check adaptation.
Recheck adaptation (AgentConfiguration) to verify if the configured OID and
SNMP version is in accordance with the managed element MIB.
Go to
/var/opt/oss/global/adaptations/archive/<adaptation_name>/co
ntent and open the SNMP FM fragment. Verify the details under
<agentSupervision> tag.

OS90538EN15GLA0
38
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

2.3 XoH Troubleshooting Cases
2.3.1 Alarms are uploaded but NetAct Monitor shows a time-
out error
When alarms are uploaded, there is an Alarm Upload Failure error, and NetAct
Monitor shows timeout.
Solution
Change the time-out value accordingly by performing the following steps:
1. Log in to the fm_pipe VM as the root user:
2. Modify the file:
$PRODUCTROOT/conf/Pref_system.interface.fmSouthMediation.xoh
.xml
3. On the node synchronization-as, edit the value of initialResponseTimeout
and responseTimeout to an appropriate value.
4. Save the changes and wait for several minutes to take effect.

OS90538EN15GLA0
39
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Troubleshooting Core Network
XoH Mediation

Alarms are uploaded but NetAct Monitor shows a time-out error
When alarms are uploaded, there is an Alarm Upload Failure error, and NetAct
Monitor shows timeout.

Solution
Change the time-out value accordingly by performing the following steps:

1. Log in to the fm_pipe VM as the root user:
2. Modify the file:
$PRODUCTROOT/conf/Pref_system.interface.fmSouthMediation.xoh.xml
3. On the node synchronization-as, edit the value of initialResponseTimeout
and responseTimeout to an appropriate value.
4. Save the changes and wait for several minutes to take effect

1 © Nokia Solutions and Networks 2015

Fig. 12

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OS90538EN15GLA0
. . . . .
40
. . . . . . . . . . . . . . . . . . . . . . . . . . . . © Nokia
. .Solutions
. .and.Networks
. . 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

2.3.2 XoH mediation reports alarms in Monitor
If Monitor is showing the following alarms coming from PLMN-PLMN/NETACT-
<cluster_ID>/VM-<XOH_NODENAME>/JBISERVER-xoh/MEDIATION-xoh,
please check the possible reason respectively as explained in the table below:.

Alarm Problem Solution
20002-Incorrect XoH mediation gets Check if the credentials are changed on
credentials FTP/SFTP credentials from the network element and the credentials in
NEAC service to connect to NEAC are synchronized.
a network element. But if
the credential is wrong, XoH
mediation sends this alarm
for the specific network
element.
20003- XoH mediation uses SFTP Check:
Certificate error and HTTPS protocols to 1. if the network element’s public RSA
communicate with a key is added in
network element if the /opt/oss/NSNxoh/smx/mf-
network element is conf/xoh-certs/known_hosts.
configured to use security
mode. But if the certificates 2. if the network element’s SSH
are not correct, XoH certificate and private key are
mediation sends this alarm installed on the network element.
for the specific network
3. if the network element’s SSH
element.
certificate or CA is imported to NetAct
under /opt/oss/NSNxoh/smx/mf-
conf/xoh-
certs/truststore.jks.
4. if the network element’s SSH
certificate or CA expires.
5. if the NetAct’s SSH certificate and
private key is imported to
/opt/oss/NSN-xoh/smx/mf-
conf/xoh-certs/keystore.jks
and its alias in keystore.jks is
XoH.
6. if the NetAct’s certificate is trustedon
by the network element if the NetAct’s
SSH certificate or CA expires.
20005- XoH mediation Check:
Configuration communicates with a 1. if the object of a network element
error network element only when is created in Object Explorer.
the integration is correctly
done. But if the integration 2. if the XoH object and FTP object
is not completed, XoH are created under the network
mediation sends this alarm

OS90538EN15GLA0
41
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

for the specific network element
element
3. object in Object Explorer.
4. if the properties of host name, port
and security mode for XoH and
FTP objects are properly filled in
Object Explorer.
5. if the network element’s
credentials are created in NEAC
20006-Unable XoH mediation connects to Check if the network element is properly
to connect NE a network element to integrated or the network is disconnected.
upload files and send SOAP
message. But if it is unable
to connect to a network
element, XoH mediation
sends this alarm for the
specific network element
20007-Internal XoH mediation gets data Check if the internal services such as
service failure from internal service. But if NASDA, WAS and NEAC work normally.
some of these internal
services are unavailable,
XoH mediation sends this
alarm for the specific
service.

OS90538EN15GLA0
42
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Troubleshooting Core Networks
XoH Mediation
Alarm Problem Solution
XoH mediation gets FTP/SFTP credentials from NEAC
Check if the credentials are changed on
20002-Incorrect service to connect to a network element. But if the
the network element and the credentials
credentials credential is wrong, XoH mediation sends this alarm for
in NEAC are synchronized.
the specific network element.
Check:
1. if the object of a network element is
created in Object Explorer.
2. if the XoH object and FTP object are
XoH mediation communicates with a network element created under the network element
20005-Configuration only when the integration is correctly done. But if the 3. object in Object Explorer.
error integration is not completed, XoH mediation sends this 4. if the properties of host name, port
alarm for the specific network element and security mode for XoH and FTP
objects are properly filled in Object
Explorer.
5. if the network element’s credentials
are created in NEAC
XoH mediation connects to a network element to upload
Check if the network element is
20006-Unable to files and send SOAP message. But if it is unable to
properly integrated or the network is
connect NE connect to a network element, XoH mediation sends this
disconnected.
alarm for the specific network element
XoH mediation gets data from internal service. But if Check if the internal services such as
20007-Internal service
some of these internal services are unavailable, XoH NASDA, WAS and NEAC work
failure
mediation sends this alarm for the specific service. normally.
1 © Nokia Solutions and Networks 2015

Fig. 13

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . .
OS90538EN15GLA0 . . . . . . . . . . . . . . . . . . . . . . . . . . .
43
. Solutions
© Nokia . . and. Networks
. . .2015. . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

OS90538EN15GLA0
44
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

3 Troubleshooting the Radio Network

OS90538EN15GLA0
45
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

3.1 NWI3 Mediation
3.1.1 Integration data upload is not working
The following objects do not appear automatically in NetAct Monitor within two
minutes after starting the integration data upload from the integration agent object
(OMS/AXC) context menu:
 NWI3-1 interface under OMS/AXC with correct nwi3systemid attribute
 The NE agent objects (RNC, WBTS, MRBTS, LNBTS) connected via the
integration agent

TIP
You can force a complete reintegration of a previously integrated OMS/AXC by
removing the NWI3-1 object from NetAct Monitor, and then start the integration data
upload from OMS/ AXC context menu.
If the NWI3-1 object exists, then the mediation skips the rest of the integration steps
and proceeds to the integration data upload.
Embedded, not standalone, AXC is a child of the WBTS object so upload the WBTS
via OMS before starting the AXC integration data upload.

Solution
1. Verify that the integration agent object (OMS/AXC) is correctly created in
NetAct Monitor.
a) Check if the correct instance and version are used.
Note:
The menu item for Integration Data Upload may be missing if the wrong version
is provided.
b) Check if the value in directIntegration is True.
2. Verify that the HTTP-1 object in NetAct Monitor is created as a child of the
integration agent (OMS/AXC).
a) • Check if the instance is 1.
b) • Check if the hostname has the correct IP value. For example,
10.11.12.13.
c) • Check if the security mode is HTTPS for OMS and HTTP for AXC.
3. Check if Mediation is running: Verify status in smanager.pl and verify the
status of Service Assemblies.

OS90538EN15GLA0
46
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

4. Check if the OMS/AXC is registered correctly.
Run nwi3_db_tool.pl in the VM node holding the nwi3 service:
[root@n83-vm09 bin]# /opt/oss/bin/nwi3_db_tool.pl -list services
SYSTEM ID | NODE | SERVICE |
TIME STAMP | STATE
------------------------------------------------------------------------
--------------------------------------
NOTIFICATION_SERVICE | n83-vm09.n83.obs.wa-slab.nsn-rdnet.net |
IDL:NWI3/NotifyPublish_V1:1.0 | 2014-04-17 19:39:33 | ACTIVE
NE-OMS-4 | n83-vm09.n83.obs.wa-slab.nsn-rdnet.net |
IDL:NWI3/SessionManager_V1:1.0 | 2014-04-23 03:51:01 | ACTIVE
------------------------------------------------------------------------
--------------------------------------
Found 2 service(s)
Check if an entry is in active state and has a timestamp equivalent to less than
24 hours.
 If the element is not registered, then the issue could be with the Network
Element Access Control(NEAC) credentials, enabling of the Centralized NE
User Management (CNUM), or with NE registration credentials
 If the element is registered but the objects do not appear in Monitor, there
might be a resource shortage on the side of the network element, or a
primary key violation in NASDA
5. Verify that NEAC contains the proper credentials.
a) In Network Element Access Control application .check if NWI3 Access
credentials have been created for the Maintenance Region (MR) or NE Type.
b) Check if the credentials have been associated to the sysop user group and
to the group of the user logged into NetAct Monitor.
c) If MR credentials are used, then check if OMS/AXC is attached to that MR.

6. Make sure that CNUM is not enabled in OMS/AXC.
To disable CNUM in OMS:
 Open the Parameter tool, and then select fsFragmentId=OMS→
omsFragmentId=System → omsFragmentId=RUIM
→omsParameterId=RUIMAutomaticActivation.
 Change the OMS Parameter Value from 1 to 0.
 Log in to the OMS or GOMS as the root user.
 Deactivate CNUM in OMS with the following command:
# fscontrolRUIM --disable

OS90538EN15GLA0
47
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

To disable CNUM in AXC:
 Log in to AXC using ftp.
 Delete the following files:
/cfg/neac.db
/var/backup/neac.db.backup
 Reboot the AXC node

7. Check that the NE registration credentials are correct.
The valid registration credentials for the network element are system/system
or nwi3system/<password>.
To acquire the nwi3system password from SCA, log in as root user in the VM
running the LDAP service with the following command:
[root@node ~]# /opt/nokia/oss/bin/syscredacc.sh -user
nwi3system -type appserv -instance appserv

 Check AXC registration credentials:
a) Open the AXC Manager.
b) Select Configuration → Management Protocol.
c) Select Configuration in Active State and click Modify.
d) Check if the username and password under Initial Registration are
valid.
 Check OMS registration credentials:
a) Open the Parameter tool using the Application Launcher.
b) Select fsClusterId=ClusterRoot → fsFragmentId=NWI3 →
fsFragmentId=mediator.
c) Select the active configuration fsnwi3N3CFId, that is, the one with
Attribute Namefsnwi3configurationActive and has an Active Value of
true.
For example, the fsnwi3N3CFId=1 object has the attribute name
fsnwi3configurationActive that has the Active Value of true.
d) Check that the values of fsnwi3initialRegistrationPassword and
fsnwi3initialRegistrationUsername are valid.

OS90538EN15GLA0
48
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

3.1.2 Alarms are not received
The active alarms in NetAct database or Monitoring tools or both do not match the
active alarms in network elements.
Solution
1. Check if the NWI3 mediation is running.
Revise nwi3 service status with smanager.pl and revise the status of the nwi3
mediation service assemblies using MF Console. Restart service or restart
service assemblies accordingly.
2. Check if the network element is properly integrated.
 Check if the NWI3-1 interface object appears with the correct nwi3stemid
attribute under the OMS/AXC object in NetAct Monitor.
If the NWI3-1 object does not appear, then start the integration data upload
from the OMS/ AXC context menu.
 Check if the network element agent objects (RNC, WBTS, MRBTS, LNBTS)
that are connected via OMS appear in NetAct Monitor.
If these do not appear, then start the integration data upload from the
OMS/AXC context menu.
 Check if the network element is correctly registered.
For more information, see Step 4 in the previous case for Integration data
upload is not working.

OS90538EN15GLA0
49
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Troubleshooting Radio Networks
NWI3 Mediation
More cases information
Alarms are not received
The active alarms in NetAct database or Monitoring tools in the course material
or both do not match the active alarms in NE.
Solution
•Check if the NWI3 mediation is running.
Revise nwi3 service status with smanager.pl and revise the status of the nwi3 mediation
service assemblies using MF Console. Restart service or restart service assemblies
accordingly.
•Check if the network element is properly integrated.
 Check if the NWI3-1 interface object appears with the correct nwi3stemid attribute
under the OMS/AXC object in NetAct Monitor.
 If the NWI3-1 object does not appear, then start the integration data upload from the
OMS/ AXC context menu.
 Check if the network element agent objects (RNC, WBTS, MRBTS, LNBTS) that are
connected via OMS appear in NetAct Monitor.
If these do not appear, then start the integration data upload from the OMS/AXC
context menu.
 Check if the network element is correctly registered.
1 Use #©/opt/oss/bin/nwi3_db_tool.pl
Nokia Solutions and Networks 2014 -list services tool to check registration of NE

Fig. 14

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OS90538EN15GLA0
. . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . © Nokia
. .Solutions
. .and.Networks
. . 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

3.1.3 PM data is not received from OMS
The OMS PM data does not reach the NetAct Performance Management database or
reporting tools or both.

Solution
1. Check if the NWI3 mediation is running.
2. Check if the network element is properly integrated.
 Check if the NWI3-1 interface object appears with the correct nwi3stemid
attribute under the OMS/AXC object in NetAct Monitor.
If the NWI3-1 object does not appear, then start the integration data upload
from the OMS/ AXC context menu.
 Check if the network element agent objects (RNC, W BTS, MRBTS, LNBTS)
that are connected via OMS appear in NetAct Monitor.
If these do not appear, then start the integration data upload from the
OMS/AXC context menu.
 Check if the network element is correctly registered.
For more information, see Step 4 in the previous case for Integration data
upload is not working
3. .Check if some measurements are active. Use Administration of Measurements
application or Element Manager accordingly to the case.
4. If the certificates are not configured for secure file transfer, then ensure that the
OMS transfer mode is HTTP instead of HTTPS.
If needed, disable secure file transfer from OMS:
a) Open the RNC OMS Parameter Tool from the Application Launcher.
b) To open the TLSMode parameter, select fsClusterId=ClusterRoot →
fsFragmentId=OMS →omsFragmentId=System → omsFragmentId=Network
→ omsFragmentId=TLS
c) Set the TLSMode(FT) parameter value to “Off”.
 Select omsFragment=TLS → omsParameterID=TLSModeFT
 Select omsParameterValue, then double-click the item under the Active
Value column to
 select Off from the drop-down menu.
5. Check the NWI3 mediation PM activity logs for any PM file transfer activity.
Check the PMEventActivity.log* and PMEventFiles.log* files in
/var/opt/oss/log/nwi3directory

OS90538EN15GLA0
51
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Troubleshooting Radio Network
NWI3 Mediation: Checking Network Element Registration

[root@n83-vm09 bin]# /opt/oss/bin/nwi3_db_tool.pl -list services
SYSTEM ID | NODE | SERVICE
| TIME STAMP | STATE
------------------------------------------------------------------
--------------------------------------------
NOTIFICATION_SERVICE | n83-vm09.n83.obs.wa-slab.nsn-rdnet.net |
IDL:NWI3/NotifyPublish_V1:1.0 | 2014-08-17 19:39:33 | ACTIVE

NE-OMS-4 | n83-vm09.n83.obs.wa-slab.nsn-rdnet.net |
IDL:NWI3/SessionManager_V1:1.0 | 2014-08-21 03:51:01 | ACTIVE
------------------------------------------------------------------
--------------------------------------------
Found 2 service(s)

1 © Nokia Solutions and Networks 2015

Fig. 15

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OS90538EN15GLA0
. . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . © Nokia
. .Solutions
. .and.Networks
. . 2015
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

3.2 Q3/MML Mediation
3.2.1 No alarm or event come from Q3 based NEs like BSC and
SGSN
When the network element is correctly integrated, yet the following operations still
fail:
 There are no NE specific alarm coming from BSC and SGSN.
 Alarm Upload, it fails.
 There is no PM data coming.
 No HWM event comes.

Solution
1. Check status of NetAct services q3user and osi using smanager.pl. Start
services if stopped.
2. Log in to theVM node where osi and q3user services are located as root user.
3. Restart the osi and q3user services like below:
[root]# /opt/cpf/bin/smanager.pl stop service osi
Service osi stopped
[root]# /opt/cpf/bin/smanager.pl start service osi
Service osi started on node <node name>
[root]# /opt/cpf/bin/smanager.pl stop service q3user
Service q3user stopped
[root]# /opt/cpf/bin/smanager.pl start service q3user
Service q3user started on node <node name>
4. If problem continues, verify that Service Assemblies in q3user mediation are
started using MF Console located in /opt/oss/NSN-nwi3/smx/mf-console

OS90538EN15GLA0
53
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

3.2.2 PM data is not received from BSC/SGSN
Solution
1. Check if the Q3 mediation is running.
 Check q3service and osi service via smanager.
 If services are not started, use smanager.pl tool to start them
 If problem continues, verify that the Service Assemblies of q3user JBI are
started. If any found stopped, start it using MF Console.
2. Check if the network element is properly integrated. Check if the OSI-1 interface
object exists with the correct attribute under the BSC/SGSN object in NetAct
Monitor. For more information, refer to both BSC and SGSN integration
documents.
3. Check if the measurements are active using Administration of Measurements
application.
4. Check if the measurement file ready event has been received by NetAct.
a) Query the following SQL statement “select * from
Q3_COMMON.Q3_PM_EVENTS where DN has the format of %BSC-xxxxxx%.
b) The event should be in the database. If not, check the
com.nsn.oss.q3.common.duplicate_event_interval value in
$Q3USER_HOME/smx/mf-conf/com.nsn.oss.q3.common.properties.
The default value is 120 minutes, which means the same file from the same
network element will be dropped within 120 minutes.
c) Check if the credential exists and is correct for “Q3 Access” for the network
element using NEAC application.
5. Check if the file has been transferred to NetAct.
a) Open download capacity log /var/tmp/NSN-q3med-
common/capacity/CL_Common_FileUpload_<date>.csv.
b) Check if the file in question has been recorded in the log with status 0
(successful status).
c) Check if the credential exists and is correct for given access type (FTP Access
or FTAM Access) in NEAC application.
d) Check if you can manually transfer the file in question by FTP client or
/opt/oss/NSNq3med-common/bin/ftam_client.sh. If the file can’t be
transferred manually, check the DCN configuration.
6. Check if the file has been decoded into OMeS.
a) open the decoding capacity log /var/tmp/NSN-q3med-
pm/capacity/CL_Pm_<bsc|sgsn>_<meas|obs>_dec_cap_<date>.csv
b) Check if the file is decoded successfully. All measurements should be
decoded into OMeS files.

OS90538EN15GLA0
54
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Troubleshooting Radio Network
Q3/MML Mediation

PM Data not received
1. Check if the Q3 mediation is running. Use smanager.pl and verify SAs using MF Console.
2. Check if the NE is properly integrated. Check if the OSI-1 interface object exists with the correct attribute
under the BSC/SGSN object in NetAct Monitor.
3. Check if the measurements are active using Administration of Measurements application.
4. Check if the measurement file ready event has been received by NetAct.
• Execute the query “select * from Q3_COMMON.Q3_PM_EVENTS where DN has the format of %BSC-xxxxxx%.
• The event should be in the database. If not, check the com.nsn.oss.q3.common.duplicate_event_interval value in
$Q3USER_HOME/smx/mf-conf/com.nsn.oss.q3.common.properties. The default value is 120 minutes, which means the
same file from the same network element will be dropped within 120 minutes.
• Check if the credential exists and is correct for “Q3 Access” for the network element using NEAC application.
5. Check if the file has been transferred to NetAct.
• Open download capacity log /var/tmp/NSN-q3med-common/capacity/CL_Common_FileUpload_<date>.csv.
• Check if the file in question has been recorded in the log with status 0 (successful status).
• Check if the credential exists and is correct for given access type (FTP Access or FTAM Access) in
NEAC application.
• Check if you can manually transfer the file in question by FTP client or /opt/oss/NSNq3med-
common/bin/ftam_client.sh. If the file can’t be transferred manually, check the DCN configuration.
6. Check if the file has been decoded into OMeS.
• open the decoding capacity log /var/tmp/NSN-q3med-
pm/capacity/CL_Pm_<bsc|sgsn>_<meas|obs>_dec_cap_<date>.csv
• Check if the file is decoded successfully. All measurements should be decoded into OMeS files
1 © Nokia Solutions and Networks 2014

Fig. 16

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . .
OS90538EN15GLA0 . . . . . . . . . . . . . . . . . . . . . . . . . . .
55
. Solutions
© Nokia . . and. Networks
. . .2015. . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mediations Troubleshooting

3.2.3 Generating upload file failed
The Upload operation from CM Operations application fails with the following error:
Generating upload file failed

Solution
1. Check the real network element version.
2. Telnet to the network element and check version by executing
ZDFD:OMU:88E;.
3. If there is difference between the versions, then modify network element version
by executing:
/ opt/oss/NSN-q3med-cm/bin/q3versMx.sh VersionChange <dn>
<version>.
For more information on q3versMx.sh script description, refer to Q3/MML
stand-alone components data flow in Administering Q3/MML Southbound
Mediation.

3.2.4 Upload or file based plan provisioning ongoing
When Upload or Provision operations are executed from CM Operations Manager
application, it fails with the following error:
UPLOAD OR FILE BASED PLAN PROVISIONING ONGOING
Solution
Directory name provided in upload request should have write-permissions for the
FTAM/FTP Unix credential user.

OS90538EN15GLA0
56
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

4 Exercises

OS90538EN15GLA0
57
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Exercise 1
Title: General Troubleshooting Tasks

Objectives: Use the general procedures for verifying the status of
Southbound and Northbound mediations

Pre-requisite: none

Task
Please execute the following exercises in the testing system using the Credentials
provided by your trainer. Answer the corresponding questions.
Query
1. From one of the mediations below, verify the corresponding status of the service
and revise the status of the corresponding Service Assemblies.
a) What commands can you use to check the status of the mediation Service?
______________________________________________________________
______________________________________________________________
______________________________________________________________
b) How can you display the Service Assemblies of your selected mediation? List
the Service Assemblies. Are any of them in a status different from "started"?
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________

c) Practice stopping /starting Service Assemblies. Select from your mediation
one Service Assembly and stop it. Check the impact of this in the status of the
corresponding service for the mediation. What can you see? Restart the
Service Assembly.
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________

OS90538EN15GLA0
58
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

d) Stop your selected mediation service. After the command is successful, check
the status of the mediation Service Assemblies. What can you see? Restart
the service after verfication.
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________

e) In which log can you find the startup/ stop of the mediation? List below the
corresponding location and related entries of the log.
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
_____________________________________________________________

f) List the logs for your selected mediation. Where are they located? What
information can you find there? Open the log files and compare with the
descriptions in Operating Documentation.
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________

OS90538EN15GLA0
59
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

g) Start tracing for your selected mediation. List the steps you need to execute
this task. Where can you find the tracer results? From the tracer result file,
what kind information is recorded by the tracer? In case of Q3/MML mediation,
select only one fragment to trace.
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________

h) Stop the trace. Verify in the log file timestamp that tracer is actually stopped.
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
_____________________________________________________________

OS90538EN15GLA0
60
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

Exercise 2
Title: Troubleshooting Radio and Core Networks

Objectives: Practice different procedures to troubleshoot specific failures
cases

Pre-requisite: none

Task
Please execute the following exercises in the testing system using the Credentials
provided by your trainer. Answer the corresponding questions.
Query
Execute the tasks for troubleshooting the case defined by your trainer. List the steps
and results of those tasks.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
___________________________________________________________________

OS90538EN15GLA0
61
© Nokia Solutions and Networks 2015
Mediations Troubleshooting

OS90538EN15GLA0
62
© Nokia Solutions and Networks 2015