AR95

New AR:
View AR | AR without Proprietary

History | States | Assignment | Timetracking
AR Text Search | More | CARES Home
Assistance Request 1-5545833

Customer Ticket N/A
Short
Etisalat UAE/CTB trace on RNC235 stopped with error while
Description
activation
Current
Jan 28 : Issue resolved after MIB rebuild.
Summary
State
Closed : Solution/Service Provided
Outage
No
Severity
Priority
Service Request Remote Technical Support

Request Type
Support
Sub-Type
Configuration Assistance
Category
Other
Sub-Category
Customer Dedicated Support
Internal
Yes
Assignment Events
Product
Product
9370 RNC
Model
9370 RNC 1500
Version
LR13.3.W
No Change Request is linked to this AR

Product Location
Site
ETISALAT ABU DHABI
Instance
9370RNC-Etisalat : In Service
Site Company
Emirates Telecommunications Corporation
City
Abu Dhabi
Country
United Arab Emirates
Contact
Name
Karim IBRAHIM ABD EL NABY
Company
Alcatel-Lucent : Open on behalf of Customer
Phone
20 127 109 0616
Karim.Ibrahim@alcatel-lucent.com
Request Method
Email
Email-Eng
Dates
Occurred
(GMT+4)
20-Jan-2015 22:09
Time now
29-Aug-2015 13:06
Reported
20-Jan-2015 22:09
AR Created
20-Jan-2015 22:37
Service Start
20-Jan-2015 22:09
Next Customer Contact
05-Feb-2015 10:30
Responded
20-Jan-2015 22:36
SA - Calculated
Respond Target
21-Jan-2015 22:09
Restored
SA - None
20-Jan-2015 22:38
Restore Target
-- --- ----
Resolved
25-Jan-2015 11:36
SA - Calculated
Resolve Target
02-Apr-2015 08:00
Targets for Restore & Resolve were extended by 11 days

to account for Pending time
Closed
04-Feb-2015 09:20
Last Modified
04-Jun-2015 16:02
Modified By
archive
Entitlement
Agreement
242895 (OXIA 627085)
Covered Service TS 24x7 (Gold, Wireless Mobile Access)

Script
This request is entitled to service.
People
Owner
TSCd-WLS-WCDMA-IN : pthota
Assignee
CloQ-WLS-WCDMA-vGlobal : pthota
Referred 1
NorP-WLS-WCDMA-vGlobal : pthota
Referred 2
TEC-MoA-WCDMA_RNC-FR : lrigault
Resolve Group
TEC-MoA-WCDMA_RNC-FR
Submitter
glipski
Description
Attachments
Etisalat UAE/CTB trace on RNC235 stopped with error while activation
Showing Investigation and Proprietary logs together in time order

separately
Show
Investigation Log and Proprietary Log

1. 20-Jan-2015 22:46
pthota
Update to Current Summary: Jan 21 : Under Investigation
2. 20-Jan-2015 22:49
pthota
Update to Current Summary: Jan 21 : Awaiting Receive Log Files / Technical

Info
3. 20-Jan-2015 23:04
pthota
ALCATEL-LUCENT PROPRIETARY
Karim (Karim)** CTR ** IBRAHIM ABD EL NABY

Workflow Status: Handled by an agent
Subject: Etisalat UAE/CTB trace on RNC235 stopped with error while
activation
Contact: Karim (Karim)** CTR ** IBRAHIM ABD EL NABY
From: "IBRAHIM ABD EL NABY, Karim (Karim)** CTR **"
<karim.ibrahim@alcatel-lucent.com>
To: ALU Support <support@alcatel-lucent.com>
Cc: "OKASHA, TAHER (TAHER)" <Taher.Okasha@alcatel-lucent.com>, "ABDEL-HALIM,
SAYED (SAYED)" <Sayed.Abdel-Halim@alcatel-lucent.com>, "WAGDY MANSOUR,

AYMAN
(AYMAN)** CTR **" <Ayman.Wagdy@alcatel-lucent.com>, "ZAHABI, RAMSEY
(RAMSEY)" <ramsey.zahabi@alcatel-lucent.com>, "EL-MIDANY, AHMED (AHMED)"
<Ahmed.El-Midany@alcatel-lucent.com>
Sent on: January 20, 2015 7:08:46 PM CET
Hello,
Short Description
: CTB trace on RNC235 stopped with error while
activation,
Current Summary
CTB trace on RNC235 stopped with error while
activation, we performed RNC MIB audit and completed successfully with

attached
warning, below the error appeared while activation,
Contact Surname
Contact Given Name
Contact Phone
Contact Company
Company
Contract Number
Karim
: Ibrahim
:+971 56 354 4568
: Alcatel-Lucent
Emirates Telecommunications Corporation
: OXIA 627085
Region
: EMEA
Country
: United Arab Emirates
City
: Abu Dhabi
Site
: ETISALAT ABU DHABI
Product Instance
Severity
Priority
: RNC 9370
: 3
: 3
Product Version
: LR13.3W
Outage
: No
Internal/External:
:Internal
-Best Regards,
Karim Ibrahim
LTE/WCDMA Integration Professional
META WLS Access Network (GNE EMEA)

M(UAE): +971-56-354-4568
M(Egy): +2010-668-44-810
ONNET:
2205 5534
image001.png (15 KB)

image002.gif (1.78 KB)
RNC_RNC235_20150120113837_NC.HTML (9.74 KB)
4. 22-Jan-2015 11:09
pthota
Update to Current Summary: Jan 22 : Suspecting issue with MIB audit

inconsistencies, escalating the AR to next Level to take expert support as
there
is already CR for MIB audit fail on this RNC.
5. 22-Jan-2015 11:09
pthota
From: THOTA, Praveen-K (Praveen-K)** CTR **

Sent: Wednesday, January 21, 2015 12:17 AM
To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **
Cc: INDIA-WCDMA-CdTSC-TEAM; OKASHA, TAHER (TAHER); ABDEL-HALIM, SAYED
(SAYED);
WAGDY MANSOUR, AYMAN (AYMAN)** CTR **; ZAHABI, RAMSEY (RAMSEY); EL-MIDANY,
AHMED
(AHMED)
Subject: AR 1-5545833//Minor//9370 RNC -235//Emirates Telecommunications
Corporation//UAE-Abudhabi//CTB trace on RNC235 stopped with error while
activation
Hi Karim,
To check the reported problem please provide the below logs.
Please provide below files from WMS
"
Files from /opt/nortel/logs/wom: wom_configurationManager-00110.*
"
Files from /opt/nortel/logs/wtk :
JobHistory-00114_<YYYMMDD>.log and
WTK_JOBSERVER-00114.*
"
Files from /opt/nortel/data/cmXML/SystemWO/ct_*
"
Files from /opt/nortel/logs/utran: cp_utran-20000.*
wms_cm_mediationserver-20001.* and dump/_RNC.userLabel_<RNC name>/xge.dump

"
Files from /opt/nortel/logs/utran: WOM_COMM_SEPE_RNC-20010.*
"
Files from /opt/nortel/logs/utran/RncProxy :
build_log_xxx_xxxx_xx_xx_xx_xx_xx.log
Please provide below files from RNC
"
:
Files from /CPx/OMUx/local/log/bos/sc (X is Active CP and Active OMU)
blogOMUx.xxxxxxxxTxxxxxx.tmp file with NACK Error time stamp.
Thanks and Regards,

Praveen Thota
+91 96899 18050
From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

To: THOTA, Praveen-K (Praveen-K)** CTR **
(SAYED);
AHMED
(AHMED)
Subject: RE: AR 1-5545833//Minor//9370 RNC -235//Emirates Telecommunications
activation
Hi Praveen,
Kindly find
all required logs
@/srv/data202256/data/server/Traces/from_france/ETISALAT_UAE/AR 1-5545833 ,
server
details below,
Please note that we already

"the
have Audit fail on RNC235 because the parameter
parameter amountRep1A under RNC235 RadioAccessService/0 DedicatedConf/0

MeasurementConfClass/0 FullEventRepCritShoMgt/0
FullEventRepCritShoMgtEvent1A/0
needs to be changed from infinity to 16 for the Audit to pass" but it's
known
issue for us respect to old ticket opened before,
When yesterday we modified the parameter and did audit to the RNC, the audit
completed with the attached warning which all related to CTB traces,
I just mention the audit issue as a note for you but it's not the main
target of
the AR, we need to know the RC why the CTB stopped with error while
activation& the corrective action of this issue,
Waiting your feedback , thanks to treat it as urgent as we need to solve the

issue
before receiving any escalation from the customer,
Server details:
SFTP://172.25.80.47
user: ftraces
password: Elv24trf
Thanks,
-BR,
Karim

(SAYED);
AHMED
(AHMED)
activation
Hi Karim,
I am checking for the NACK message sent by RNC when the trace was stopped
with
Errors.
"NACK happened when RNC does not like something in the MIB and does not know
what
to do with it so RNC send a NACK message to OAM when mib build fail"
As per the Image the CTB trace was started on 18-01-2015 03:48AM. But I am
not
finding the logs for that date in RNC or WMS. So please check for the
corresponding date logs or start the ctb trace again and request you to
provide
the required all logs again. Thanks.
Thanks and Regards,

Praveen Thota
+91 96899 18050

(SAYED);
AHMED
(AHMED)
activation
Hi Karim,
Apart from Logs I may need some more information to understand the issue
better.
1)
Please confirm whether this is the first time CTB trace was started on
this RNC
on 18th Jan?
No , the CTB trace was working fine before, also a week before we
launched a CTB
trace on the RNC normally but the output trace files was empty!.
2)
CTB trace was started on 18th Jan and it stopped with errors? Whether
it has
been tried again recently on this RNC?
Yes I tried to it more times with different IMSIs but

same error,
3)
I received the
Any changes done on RNC before trace start time or preceding days?
@ 18th Jan we had activity to add 2 new MSC to the RNC, but we already
applied
the WOs before this date about 3 weeks, Also I performed @ 18th Lp/0
switchover as
the new MSCs Assoc was up but the signaling down (CS new components were
down), so
switchover done the components come Up,
4)
As per below the parameter change happened on 20th Jan and after that
Audit
completed with warnings, before it was okay and there was no issue in audit
when
CTB trace was started?
The Audit failure is exiting on the RNC after LR13.3 intra upgrade and
it was
recommend to change the mentioned parameter below to this value to enhance
the
KPIs, so if we need to make Audit anytime it will fail except of we change
the
parameter , so I changed ot to try the MIB audit and see if it will give any
new
errors or warning then I reverted the parameter again to the recommend
value,
5)
Please provide any other details if you have related to this issue.
Waiting your feedback.

Thanks and Regards,
Praveen Thota
+91 96899 18050

Sent: Wednesday, January 21, 2015 2:32 PM
(SAYED);
AHMED
(AHMED)
activation
Hello Praveen,
Kindly find my comments highlighted below,
Waiting your feedback,

Thanks,
-BR,
Karim

(SAYED);
AHMED
(AHMED)
activation
Hi Karim,
As per below there is some traces exist in WMS, Please confirm all traces
are
deleted and there is no trace active from WMS NSP.
>: utran list TraceSession
{TraceSession RNC235_UMAINSERV1} {TraceSession trial_UMAINSERV1}
Thanks and Regards,

Praveen Thota
+91 96899 18050

(SAYED);
AHMED
(AHMED)

activation
Hello Praveen,
Do you mean that there are a hanging trace session? From NSP no active CTB
traces
on
RNC235.
Thanks,
-BR,
Karim

(SAYED);
AHMED
(AHMED)
activation
Hi Karim,
Yes, there is a trace session hanging for RNC235 (TraceSession

RNC235_UMAINSERV1).
Please try to delete it through WICL and launch MIB audit again and share
the
results.
If TraceSession trial_UMAINSERV1 is completed then please delete this as

well from
NSP or WICL.
Thanks and Regards,

Praveen Thota
+91 96899 18050

(SAYED);
AHMED
(AHMED)
activation
Hello Praveen,
Please note that I performed list check using WICL and only TraceSession
trial_UMAINSERV1 is the existing one, I tried to delete it as below but the
command failed, can you check,
....wait please
Wireless Internet Command Language (WICL)
>: utran list TraceSession
{TraceSession trial_UMAINSERV1}
>: delete TraceSession >trial
wrong # args: should be "::wisci::manipulationServices::delete domain server

mod
ule internalDN args"
>: delete TraceSession> trial
wrong # args: should be "::wisci::manipulationServices::delete domain server

module internalDN args"
Thanks,
-BR,
Karim


(SAYED);
AHMED
(AHMED)
activation
Hi Karim,
As per MOC conversation, Now there is no TraceSessions. Please perform MIB

audit
and share the results. Also provide logs when you run CTB again in this RNC.
Thanks and Regards,

Praveen Thota
+91 96899 18050

(SAYED);
AHMED
(AHMED)
activation
Hi Praveen,
Yes I deleted the TraceSession trial_UMAINSERV1 and I didn't find the

TraceSession
RNC235_UMAINSERV1 , as discussed we performed the parameter change and
pefromed
Audit, the same warnings still exist as attached, also we tried to do
another CTB
trace and stopped with error.
We terminated the session but as below the session still exist on utrantrace
session, Please advise, attached the required files,
>:
utran list TraceSession
{TraceSession Test_UMAINSERV1} {TraceSession test_UMAINSERV1}
Thanks,
-BR,
Karim

Sent: Thursday, January 22, 2015 11:12 AM
(SAYED);
AHMED
(AHMED)
activation
Hi Praveen,
The RNClogcollection uploaded to the same mentioned path below,
Thanks,
-BR,
Karim
6. 22-Jan-2015 12:38
pthota
Escalating this AR to TEC for support, suspecting issue with MIB audit
inconsistencies.
1 - DESCRIPTION TEMPLATE
SUMMARY: RNC235 CTB trace stopped with Errors with message "RNC/RNC235'
rejects
the command with a NACK : TgeAckEvent ( neId=183422173, refPe=35021)
Cause: CAUS_NS"
Concerned RNC: RNC235
Concerned component (card, LP, OMU...): None
Type of alarm: CTB trace stopped with Errors
Time of the issue: 2015/01/21-17:03

Impact (partial, full outage on RNC...): No Outage, Not able to run CTB
trace.
Product release: LR 13.3
Patch release:
Reproducible: Yes
Suggested solution (workaround...): Suggested to remove the warnings in MIB
Audit.
PROBLEM DETAILS:
1. Action performed (feature activation, upgrade, work-order
application...): None
2. System behavior (high-level + detailed description if relevant): No
Impact
other than MIB audit warnings and CTB trace issue.
3. TSC/TIS analysis: When verified through wicl, found some trace sessions
which
were not present in NSP. deleted them manually with Wicl and performed
MIB audit, but again condition was same. In MIB audit we have observed
warnings
related to Access session but when we verified in snapshot those parameters
are not exist in RNC. So unable to correct them. we suspect CTB trace
stopped with
Errors due to the warnings of MIB audit. these warnings are pointed to
Access session and issue is with CTB trace only. CTG is not having any issue
as
per AR requestor.
4. Reason for priority/severity choice: Customer want CTB trace to be
enabled for
some other issue and hence customer pressure is there.
5. Contact information: Praveen Kumar Thota +91 9689918050
CONFIGURATION:
1. Hardware configuration: RNC 9370 with 12 DCPS
2. Software configuration: LR 13.3
IMPACT:
1. Frequency of the problem (once, x times, sporadic, systematic...): None
2. Features impacted (iMCTA, CAC, Rate Adaptation, HSxPA over IUR,
other...): None
3. Impact (call drop, board reset, KPI degradation, redundancy...): Unable
to run
CTB trace.
HISTORY:
1. Which & how many NEs impacted: None
2. How the issue was detected by customer or local ALU support (via KPIs
degradation, customer complaint...): Local ALU team.
3. When the issue has started: Not Sure
4. What actions were performed before or in time when the issue begun
(feature
activation, upgrade, 3rd party actions...) and is the issue
still present: No actions as per requestor and now issue is present.
5. What recovery actions have been done & with what results: removed the
trace
sessions through wicl and tried again with no success.
MIB rebuild not tried since there is inconsistencies in MIB audit. we need
TEC
confirmation to proceed with MIB rebuild after checking the logs.
2 - LOGS AND TRACES are under path :

/srv/data202256/data/server/Traces/from_france/ETISALAT_UAE/AR 1-5545833
1. RNCLogCollection => collected and uploaded on server :

LogCollection_RNC235_full_20150121_194536.tar.gz.
2. MIB audit result is uploaded : AR_1-5545833_Logs_21012015.rar.
3. network snapshot.
2. Below logs are collected and uploaded on France server : file name AR_1-5545833_Logs_21012015.rar.
files from WMS

o
00110.*
Files from /opt/nortel/logs/wom: wom_configurationManager-
o
Files from /opt/nortel/logs/wtk :
00114_<YYYMMDD>.log
JobHistory-
and WTK_JOBSERVER-00114.*
o
Files from /opt/nortel/data/cmXML/SystemWO/ct_*
Files from /opt/nortel/logs/utran: cp_utran-20000.*
wms_cm_mediationserver-20001.* and dump/_RNC.userLabel_<RNC name>/xge.dump

o
Files from /opt/nortel/logs/utran: WOM_COMM_SEPE_RNC-20010.*
Files from /opt/nortel/logs/utran/RncProxy :
build_log_xxx_xxxx_xx_xx_xx_xx_xx.log
files from RNC

o
OMU)
:
Files from /CPx/OMUx/local/log/bos/sc (X is Active CP and Active
blogOMUx.xxxxxxxxTxxxxxx.tmp file
History : There is already one AR related to MIB audit failed for this
RNC235
(1-5351273), Later CR created for the same., GPS provided WA as one WO to
run
before starting the Audit
to make it successful. attached that WO as well.
Please let us know if any other Logs required. Thanks.

Time Tracking Entry Added - IR01-TSA3
7. 23-Jan-2015 15:05
lrigault
Lab investigation result:
Futher lab retest on AR 1-5351273, provided another work-around to solve the

MIB
audit failure for "the parameter amountRep1A under RNC235
RadioAccessService/0
DedicatedConf/0 MeasurementConfClass/0 FullEventRepCritShoMgt/0
FullEventRepCritShoMgtEvent1A/0 needs to be changed from infinity to 16 for
the
Audit to pass".
to
Currently the parameter value is set to infinity in WMS and
propagate to the RNC then MIB rebuild can be used to update this parameter
with
"change not implemented ON LINE" (CR 1356516).
We tested CTB creation with such MIB audit failure in our lab and CTB IMSI
creation was successful.
The issue of CTB creation is not related to this MIB audit failure.
Current logs analysis from the customer site

++++++++++
It is found OMU log warning for each occurence of CTB IMSI creation failure
"Object Trca_SessionCTb_inst already created" and such MIB audit execution

result
with 6 inconsistencies.
File: "utran/AuditResult/RNC_RNC235_20150120113837_NC.xml" target="XML_AUDIT

Will clearly report object diffMissingOmcData on RNC/0 CallTrace/0
AccessSession/0
|-logs
|
+-log[0]
|-id
-> 0
|-severity
-> 0
|-msg
-> Begin Audit on 'RNC/RNC235'
+-log[1]
|-id
-> 1
|-severity
-> 1
|-msg
-> Verdict NOT consistent, found 6 inconsistencies
+-log[2]
|-id
-> 2
|-severity
-> 0
|-msg
-> <a
href="utran/AuditResult/RNC_RNC235_20150120113837_NC.xml"
target="XML_AUDIT">
|
+-log[3]
|-id
-> 3
|-severity
-> 0
|-msg
-> End Audit on 'RNC/RNC235' succeeded
|-startTime
-> 2015/01/20-11:29:42,851+04:00
|-endTime
-> 2015/01/20-11:38:38,543+04:00
2015/01/20-11:13:52,442+04:00 | ItMedServUtilities.cc:255 | Thr:23 | EXT |

component-name | class-name | CId:13473713.9268
extern: DN impacted for TGE 0 = {
, RNC/RNC235 CallTrace/0 AccessSession/0, }(number of elements=1)
18/01/2015 03:48:03.284 num:0000117b adm_umts_trca_p.c.766 [0x6c458000]

CM_CA @
OMU_1(Lp/5,Ap/5) (PERM): Adm:Object Trca_SessionCTb_inst already created.
18/01/2015 04:48:59.612 num:000015e5 adm_umts_trca_p.c.766 [0x6c458000]
CM_CA @
20/01/2015 11:13:52.426 num:0000c3ec adm_umts_trca_p.c.766 [0x6c458000]
CM_CA @
20/01/2015 11:28:18.328 num:0000c4e6 adm_umts_trca_p.c.766 [0x6c458000]
CM_CA @

CM_CA @
18/01/2015 03:48:03.284 num:0000117c adm_umts_trca_p.c.767 [0x6c458000]
CM_CA @
OMU_1(Lp/5,Ap/5) (PERM): Adm:Can't perform CREATE operation.
18/01/2015 03:48:03.284 num:0000117d adm_std_mo_p.c.662 [0x6c458000]
@
OMU_1(Lp/5,Ap/5) (PERM): Adm -> ao update cds funct ko
processing_sessionctb_trca_Trca_SessionCTb_inst returns : 1
+++++++++++
CM_CA
WMS log for MIB audit execution report this error in file
wms_cm_mediationserver-20001.tra
<diffMissingOmcData on cd> -object:"RNC/0 CallTrace/0 AccessSession/0"
2015/01/20-11:35:54,164+04:00 | WMS_CM_ItAuditManager_i.cc:269 | Thr:23 |

EXT |
component-name | class-name | CId:13474310
end input: End Traitment: #AUDITMIB# -DN:???
<WMS_CM_ItAuditManager_i::computeNextTge> Succeeded

EXT |
begin input: Start Traitment: #AUDITMIB# -DN:???
<WMS_CM_ItAuditManager_i::computeNextTge>

EXT |
begin input: WMS_CM_ItAuditManager_i::calculateNextTge
2015/01/20-11:35:55,159+04:00 | Container.cc:49 | Thr:20 | EXT | componentname |

class-name | CId:13474310
extern: no such dn in tree "RNC/0 CallTrace/0 AccessSession/0"
2015/01/20-11:35:55,161+04:00 | WMS_CM_AuditMediation.cc:40 | Thr:20 | EXT |

extern: Traitment: #AUDITMIB# -DN:??? <diffMissingOmcData on cd>
-object:"RNC/0
CallTrace/0 AccessSession/0"
-cd:
<administrativeState>
<locked/>
</administrativeState>
<duration>240</duration>
<datafileUploadSize>524288</datafileUploadSize>
<datafileUploadPeriod>15</datafileUploadPeriod>
<externalSessionId>378</externalSessionId>
2015/01/20-11:35:55,164+04:00 | AuditMed.cc:442 | Thr:20 | EXT | componentname |

extern: Adding InconsistencyOnSMO in the diff tree: RNC/0 CallTrace/0
AccessSession/0 TraceFilter/0

AccessSession/0

extern: Adding missing OMC Object in the diff tree: RNC/0 CallTrace/0
AccessSession/0 NbapDedicatedMeasConfigForCallTrace/0
AccessSession/0 RrcDedicatedMeasConfigForCallTrace/0

AccessSession/0 UePositionMeasConfigForCallTrace/0

AccessSession/0
It is suspected such parameter was changed to make the audit successful and
report
the inconsistencies.
"the parameter amountRep1A under RNC235 RadioAccessService/0 DedicatedConf/0

needs to be changed from infinity to 16 for the Audit to pass"
|-registTime
-> 2015/01/20-11:24:22
|-userData
|-neDn
-> RNC/RNC235
|-objectDn
-> RNC/RNC235
|-sessionName
-> Online Commands
"cmXML/CMSessions/20150120/Online+Commands/RNC_RNC235_validate_2015012011242
4_0.ht
ml"
And again ON LINE command was executed a second time to set back the value
which
fail the MIB audit
|-registTime
-> 2015/01/20-11:48:42
|-userData
|-neDn
-> RNC/RNC235
|-objectDn
-> RNC/RNC235
|-sessionName
-> Online Commands
5_0.ht
ml"
++++++++++
==========
Action Plan for this AR 1-5545833 on CTB creation issue.
1)
Check and provide MIB audit result file showing the 6 inconsistencies
For last MIB audit execution on RNC235 it was completed
|-msg
+-log[2]
|-id
-> 2
|-severity
-> 0
|-msg
-> <a
target="XML_AUDIT">
|
+-log[3]
|-id
-> 3
|-severity
-> 0
|-msg
|-startTime
-> 2015/01/20-11:29:42,851+04:00
|-endTime
-> 2015/01/20-11:38:38,543+04:00
2) Backup RNC 235 MIB
Record RNC commit provisioning name with telnet session "d pr"
Select RNC, right click configuration / RNC backup
This MIB backup command execution takes several minutes.
This MIB backup could be studied afterwards if MIB rebuild was not going to
fix
the IMSI CTB creation failure.
3) Check if CTB IMSI session are already created for that RNC in WMS
database
WICL
>: RNC/RNC235 CallTrace/0 list AccessSession
?
On all listed session executed display command, for instance here
"AccessSession
0" instance.
>: RNC RNC235 CallTrace 0 AccessSession 0 display
If RNC RNC235 CallTrace 0 AccessSession 0 exist in WMS then delete it with

the
call trace wizzard then you should be able to create the new one you expect.
++++++++
If RNC RNC235 CallTrace 0 AccessSession 0 does NOT exist in WMS then execute
RNC
MIB rebuild to clean the already created instance in the RNC.
You could temporary change change back the parameter value to 16 to get the
MIB
audit not failed anymore. This might indicate with the MIB completion the
inconsistency on object "RNC RNC235 CallTrace 0 AccessSession 0 " being
present in
RNC MIB but not WMS database from the MIB audit result.
This seems what was already executed on site on 20th Jan 2015 (two ON
LINE
command execution previously listed)
++++++++
4) Execute impacted RNC 235 MIB rebuild from WMS

5) Run again MIB and check it is successful
6) Create CTB IMSI session and check it is successful.
7) In case this action is not succesful , please provide:
- this RNC backup (from /opt/nortel/data/utran/neBackup)
- jobHistory file from /opt/nortel/wtk
- RncFullLogCollection.
Here example for lab RNC73

/opt/nortel/data/utran/NEImage/RNC/RNC73
RNC73_20150123_101244.tar.gz
==========
8. 25-Jan-2015 13:17
pthota
Update to Current Summary: Jan 25 : provided TEC update to the site team,
waiting
for their feedback.
9. 28-Jan-2015 09:55
pthota
Update to Current Summary: Jan 28 : provided TEC update to the site team,
waiting
for their feedback.
10. 04-Feb-2015 09:19

pthota
Update to Current Summary: Jan 28 : Issue resolved after MIB rebuild.
11. 04-Feb-2015 09:19

pthota

Sent: Sunday, January 25, 2015 1:16 PM
(SAYED);
AHMED
(AHMED)
activation
Hi Karim,
Please find below TEC update on this issue.
Lab investigation result:

Futher lab retest on AR 1-5351273, provided another work-around to solve the
MIB
audit failure for "the parameter amountRep1A under RNC235
RadioAccessService/0
the
Audit to pass".
to
Currently the parameter value is set to infinity in WMS and
propagate to the RNC then MIB rebuild can be used to update this parameter
with
"change not implemented ON LINE" (CR 1356516).
We tested CTB creation with such MIB audit failure in our lab and CTB IMSI
creation was successful.
The issue of CTB creation is not related to this MIB audit failure.
Current logs analysis from the customer site

++++++++++
It is found OMU log warning for each occurence of CTB IMSI creation failure
"Object Trca_SessionCTb_inst already created" and such MIB audit execution
result
with 6 inconsistencies.
File: "utran/AuditResult/RNC_RNC235_20150120113837_NC.xml" target="XML_AUDIT

Will clearly report object diffMissingOmcData on RNC/0 CallTrace/0
AccessSession/0
|-logs
|
+-log[0]
|-id
-> 0
|-severity
-> 0
|-msg
-> Begin Audit on 'RNC/RNC235'
+-log[1]
|-id
-> 1
|-severity
-> 1
|-msg
+-log[2]
|-id
-> 2
|-severity
-> 0
|-msg
-> <a
target="XML_AUDIT">
|
+-log[3]
|-id
-> 3
|-severity
-> 0
|-msg
|-startTime
-> 2015/01/20-11:29:42,851+04:00
|-endTime
-> 2015/01/20-11:38:38,543+04:00
2015/01/20-11:13:52,442+04:00 | ItMedServUtilities.cc:255 | Thr:23 | EXT |

component-name | class-name | CId:13473713.9268
extern: DN impacted for TGE 0 = {
, RNC/RNC235 CallTrace/0 AccessSession/0, }(number of elements=1)

CM_CA @
18/01/2015 04:48:59.612 num:000015e5 adm_umts_trca_p.c.766 [0x6c458000]
CM_CA @
20/01/2015 11:13:52.426 num:0000c3ec adm_umts_trca_p.c.766 [0x6c458000]
CM_CA @
20/01/2015 11:28:18.328 num:0000c4e6 adm_umts_trca_p.c.766 [0x6c458000]
CM_CA @

CM_CA @
18/01/2015 03:48:03.284 num:0000117c adm_umts_trca_p.c.767 [0x6c458000]
CM_CA @
OMU_1(Lp/5,Ap/5) (PERM): Adm:Can't perform CREATE operation.
18/01/2015 03:48:03.284 num:0000117d adm_std_mo_p.c.662 [0x6c458000]
@
OMU_1(Lp/5,Ap/5) (PERM): Adm -> ao update cds funct ko
processing_sessionctb_trca_Trca_SessionCTb_inst returns : 1
CM_CA
+++++++++++
WMS log for MIB audit execution report this error in file
wms_cm_mediationserver-20001.tra
<diffMissingOmcData on cd> -object:"RNC/0 CallTrace/0 AccessSession/0"

EXT |
end input: End Traitment: #AUDITMIB# -DN:???
<WMS_CM_ItAuditManager_i::computeNextTge> Succeeded

EXT |
begin input: Start Traitment: #AUDITMIB# -DN:???
<WMS_CM_ItAuditManager_i::computeNextTge>

EXT |
begin input: WMS_CM_ItAuditManager_i::calculateNextTge
2015/01/20-11:35:55,159+04:00 | Container.cc:49 | Thr:20 | EXT | componentname |

extern: no such dn in tree "RNC/0 CallTrace/0 AccessSession/0"
2015/01/20-11:35:55,161+04:00 | WMS_CM_AuditMediation.cc:40 | Thr:20 | EXT |

extern: Traitment: #AUDITMIB# -DN:??? <diffMissingOmcData on cd>

-object:"RNC/0
CallTrace/0 AccessSession/0"
-cd:
<administrativeState>
<locked/>
</administrativeState>
<duration>240</duration>
<datafileUploadSize>524288</datafileUploadSize>
<datafileUploadPeriod>15</datafileUploadPeriod>
<externalSessionId>378</externalSessionId>

AccessSession/0 TraceFilter/0

AccessSession/0

AccessSession/0 NbapDedicatedMeasConfigForCallTrace/0

AccessSession/0 RrcDedicatedMeasConfigForCallTrace/0

AccessSession/0 UePositionMeasConfigForCallTrace/0

AccessSession/0
It is suspected such parameter was changed to make the audit successful and
report
the inconsistencies.
"the parameter amountRep1A under RNC235 RadioAccessService/0 DedicatedConf/0

needs to be changed from infinity to 16 for the Audit to pass"
|-registTime
|-userData
|-neDn
-> 2015/01/20-11:24:22
-> RNC/RNC235
|-objectDn
-> RNC/RNC235
|-sessionName
-> Online Commands
4_0.ht
ml"
And again ON LINE command was executed a second time to set back the value
which
fail the MIB audit
|-registTime
-> 2015/01/20-11:48:42
|-userData
|-neDn
-> RNC/RNC235
|-objectDn
-> RNC/RNC235
|-sessionName
-> Online Commands
5_0.ht
ml"
++++++++++
==========
Action Plan for this AR 1-5545833 on CTB creation issue.
1)
Check and provide MIB audit result file showing the 6 inconsistencies
For last MIB audit execution on RNC235 it was completed
|-msg
+-log[2]
|-id
-> 2
|-severity
-> 0
|-msg
-> <a
target="XML_AUDIT">
|
+-log[3]
|-id
-> 3
|-severity
-> 0
|-msg
|-startTime
-> 2015/01/20-11:29:42,851+04:00
|-endTime
-> 2015/01/20-11:38:38,543+04:00
2) Backup RNC 235 MIB
Record RNC commit provisioning name with telnet session "d pr"
Select RNC, right click configuration / RNC backup
This MIB backup command execution takes several minutes.
This MIB backup could be studied afterwards if MIB rebuild was not going to
fix
the IMSI CTB creation failure.
3) Check if CTB IMSI session are already created for that RNC in WMS
database
WICL
>: RNC/RNC235 CallTrace/0 list AccessSession
?
On all listed session executed display command, for instance here
"AccessSession
0" instance.
>: RNC RNC235 CallTrace 0 AccessSession 0 display
If RNC RNC235 CallTrace 0 AccessSession 0 exist in WMS then delete it with

the
call trace wizzard then you should be able to create the new one you expect.
++++++++
If RNC RNC235 CallTrace 0 AccessSession 0 does NOT exist in WMS then execute
RNC
MIB rebuild to clean the already created instance in the RNC.
You could temporary change change back the parameter value to 16 to get the
MIB
audit not failed anymore. This might indicate with the MIB completion the
inconsistency on object "RNC RNC235 CallTrace 0 AccessSession 0 " being
present in
RNC MIB but not WMS database from the MIB audit result.
LINE
This seems what was already executed on site on 20th Jan 2015 (two ON
command execution previously listed)

++++++++
4) Execute impacted RNC 235 MIB rebuild from WMS

5) Run again MIB and check it is successful
6) Create CTB IMSI session and check it is successful.
7) In case this action is not succesful , please provide:
- this RNC backup (from /opt/nortel/data/utran/neBackup)
- jobHistory file from /opt/nortel/wtk
- RncFullLogCollection.
Here example for lab RNC73

/opt/nortel/data/utran/NEImage/RNC/RNC73
RNC73_20150123_101244.tar.gz
==========
Thanks and Regards,

Praveen Thota
+91 96899 18050

Sent: Sunday, January 25, 2015 3:21 PM
(SAYED);
AHMED
(AHMED)
activation
Hello Praveen,
Thanks for your feedback, So finally it's recommended to perform RNC Rebuild
to
solve this issue but from below we don't receive the RC , Why the CTB
fail? , also
for next we should change the parameter with MIB Rebuild not online?.
Thanks,
-BR,
Karim

Sent: Monday, January 26, 2015 8:29 PM
(SAYED);
AHMED
(AHMED)
activation
Hi Karim,
Please find below update for your queries........

============================================================================
======
=============================================
Yes, please execute MIB rebuild operation on this RNC235.
About the RCA on the CTB inconsistency with object not existing in the WMS
but
being present in RNC, it will be difficult to find but you can collect all
jobHistory files (JobHistory-00114-yyyymmdd) from /opt/nortel/logs/wtk (grep
command on callTrace object could be used to check what might have

happened). If
issue occurred more than one one month ago then we will not have logs for
that. It
is advised to perform at least on successful MIB audit (followed by dbbackup
+
image backup) every two weeks or every month (but every month is a limit
with risk
of lising jobHistory files information limited to one month backlog).
Now about the CR 1356516 and old AR 1-5351273 (2 options are proposed).
It was advised to get the MIB audit successful (as initially reported by one
site
team).
"the parameter amountRep1A under RNC235 RadioAccessService/0
the
Audit to pass".
As on site team seems to insist on using the parameter value "infinity"

intead of
value "16", i checked in our lab what kind of additional work-around could
be
used. CR owner reported the SET ONLINE command is not implemented until
LR13.3.B.
But this parameter can be updated with MIB rebuild as verified in our lab.
This is
provide full answer to on site team wish of using "infinity" parameter value
and
get the MIB audit successful. It is not advised at all to keep the MIB audit
always failed because MIB audit inconsistency will not be detected
immediately.
============================================================================
======
=============================================
Thanks and Regards,

Praveen Thota
+91 96899 18050

Sent: Thursday, January 29, 2015 5:17 AM
(SAYED);
AHMED
(AHMED); BERIDY, AHMED (AHMED)
activation
Hello Praveen,
Kindly note that the RNC235 rebuild done , Audit after is OK without warning
or
erros as attached also CTB trace is working fine.
Please note we back the parameter again to infinity in order don't have any
KPIs
degradation,
we will plan for RNC audit every 2 weeks,
Will keep you updated if we face any issue related to this issue.
Thanks,
-BR,
Karim

Sent: Tuesday, February 03, 2015 12:01 PM
(SAYED);
AHMED
(AHMED); BERIDY, AHMED (AHMED); MERAD, ABDESSAMAD (ABDESSAMAD)
activation
Hello Praveen,
Thanks to close the AR, the issue solved as I mentioned below, and I tested
again
the Audit &CTB and both OK.
Thanks,
-BR,
Karim
M: +971 (0)56-354-4568

Sent: Wednesday, February 04, 2015 10:36 AM
(SAYED);
AHMED
(AHMED); BERIDY, AHMED (AHMED); MERAD, ABDESSAMAD (ABDESSAMAD)
activation
Hi Karim,
Thanks for AR closure confirmation. We are closing this AR.
Thanks and Regards,

Praveen Thota
+91 96899 18050
Time Tracking Entry Added - IR01-TSA1
Resolution
issue solved after MIB rebuild of the RNC.
End of Assistance Request 1-5545833
New AR:
View AR | AR without Proprietary | OLCS

History | States | Assignment | Timetracking
AR Text Search | More | CARES Home

AR95

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

AR95

Uploaded by

Copyright:

Available Formats

New AR:

View AR | AR without Proprietary

Assistance Request 1-5545833

Etisalat UAE/CTB trace on RNC235 stopped with error while

Jan 28 : Issue resolved after MIB rebuild.

Closed : Solution/Service Provided

Service Request Remote Technical Support

Customer Dedicated Support

9370 RNC 1500

No Change Request is linked to this AR

ETISALAT ABU DHABI

Emirates Telecommunications Corporation

United Arab Emirates

Karim IBRAHIM ABD EL NABY

Alcatel-Lucent : Open on behalf of Customer

Targets for Restore & Resolve were extended by 11 days

242895 (OXIA 627085)

Covered Service TS 24x7 (Gold, Wireless Mobile Access)

This request is entitled to service.

Etisalat UAE/CTB trace on RNC235 stopped with error while activation

Showing Investigation and Proprietary logs together in time order

Investigation Log and Proprietary Log

Update to Current Summary: Jan 21 : Under Investigation

Update to Current Summary: Jan 21 : Awaiting Receive Log Files / Technical

Karim (Karim)** CTR ** IBRAHIM ABD EL NABY

SAYED (SAYED)" <Sayed.Abdel-Halim@alcatel-lucent.com>, "WAGDY MANSOUR,

: CTB trace on RNC235 stopped with error while

CTB trace on RNC235 stopped with error while

activation, we performed RNC MIB audit and completed successfully with

warning, below the error appeared while activation,

Contact Given Name

:+971 56 354 4568

Emirates Telecommunications Corporation

: United Arab Emirates

: ETISALAT ABU DHABI

META WLS Access Network (GNE EMEA)

image001.png (15 KB)

Update to Current Summary: Jan 22 : Suspecting issue with MIB audit

is already CR for MIB audit fail on this RNC.

From: THOTA, Praveen-K (Praveen-K)** CTR **

To check the reported problem please provide the below logs.

Please provide below files from WMS

Files from /opt/nortel/logs/wom: wom_configurationManager-00110.*

Files from /opt/nortel/logs/wtk :

Files from /opt/nortel/data/cmXML/SystemWO/ct_*

Files from /opt/nortel/logs/utran: cp_utran-20000.*

wms_cm_mediationserver-20001.* and dump/_RNC.userLabel_<RNC name>/xge.dump

Files from /opt/nortel/logs/utran: WOM_COMM_SEPE_RNC-20010.*

Files from /opt/nortel/logs/utran/RncProxy :

Please provide below files from RNC

Files from /CPx/OMUx/local/log/bos/sc (X is Active CP and Active OMU)

blogOMUx.xxxxxxxxTxxxxxx.tmp file with NACK Error time stamp.

Thanks and Regards,

From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

all required logs

Please note that we already

have Audit fail on RNC235 because the parameter

parameter amountRep1A under RNC235 RadioAccessService/0 DedicatedConf/0

Waiting your feedback , thanks to treat it as urgent as we need to solve the

From: THOTA, Praveen-K (Praveen-K)** CTR **

Thanks and Regards,

From: THOTA, Praveen-K (Praveen-K)** CTR **

Yes I tried to it more times with different IMSIs but

Waiting your feedback.

Karim (Karim) CTR IBRAHIM ABD EL NABY

From: THOTA, Praveen-K (Praveen-K) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: THOTA, Praveen-K (Praveen-K) CTR

From: THOTA, Praveen-K (Praveen-K) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: THOTA, Praveen-K (Praveen-K) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: THOTA, Praveen-K (Praveen-K) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: THOTA, Praveen-K (Praveen-K) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR