Assistance Request 1-5522436: View AR

New AR:
View AR | AR without Proprietary

History | States | Assignment | Timetracking
AR Text Search | More | CARES Home
Assistance Request 1-5522436

Customer Ticket n/a
Short
its
TEC support needed to check the RC of this issue, and if
Description
sever
affecting the KPIs or not, the network deployed with PTP

for Sync.
Current
Closure approval granted
Summary
State
Closed : Solution/Service Provided
Outage
No
Severity
Priority
Service Request Remote Technical Support

Request Type
Support
Sub-Type
Diagnose
Category
Software
Internal
Yes
Assignment Events
Product
Product
9926 DBS-WCDMA
Version
LR13.1W
No Change Request is linked to this AR

Product Location
Site
ETISALAT ABU DHABI
Instance
9926DBS WCDMA-Etisalat : In Service
Site Company
Emirates Telecommunications Corporation
City
Abu Dhabi
Country
United Arab Emirates
Contact
Name
Karim IBRAHIM ABD EL NABY
Company
Alcatel-Lucent : Open on behalf of Customer
Phone
20 127 109 0616
Karim.Ibrahim@alcatel-lucent.com
Request Method
Email
Email-CR
Dates
Occurred
(GMT+4)
06-Jan-2015 10:27
Time now
29-Aug-2015 13:21
Reported
06-Jan-2015 10:27
AR Created
06-Jan-2015 10:33
Service Start
06-Jan-2015 10:27
Next Customer Contact
07-Jan-2020 03:00
Responded
06-Jan-2015 23:00
SA - Calculated
Respond Target
07-Jan-2015 10:27
Restored
SA - None
23-Jan-2015 18:20
Restore Target
-- --- ----
Resolved
23-Jan-2015 18:27
SA - Calculated
Resolve Target
24-Mar-2015 03:15
Targets for Restore & Resolve were extended by 16 days

to account for Pending time
Closed
23-Jan-2015 18:29
Last Modified
24-May-2015 16:05
Modified By
archive
Entitlement
Agreement
242895 (OXIA 627085)
Covered Service TS 24x7 (Gold, Wireless Mobile Access)

Script
This request is entitled to service.
People
Owner
TSCr-MoA-WCDMA-SK : rories
Assignee
CloQ-WLS-WCDMA-vGlobal : unassigned
Referred 1
NorP-WLS-WCDMA-vGlobal : rories
Resolve Group
NorP-WLS-WCDMA-vGlobal
Submitter
ansikors
Description
Attachments
From: "IBRAHIM ABD EL NABY, Karim (Karim)** CTR **"

<karim.ibrahim@alcatel-lucent.com>
Sent: 2015-01-06 7:27:12
To: ALU Support <support@alcatel-lucent.com>
Subject: Repeated alarm
"SYN NOT locked to the Network"
Hello,
Short Description
Repeated PTP alarm fail over the network,
on behalf of the alarm SYN NOT locked to the Network
Current Summary
: TEC support needed to check the RC of this
issue, and if its affecting the KPIs or not, the network deployed with PTP
sever for Sync.
Contact Surname
Contact Given Name
Contact Phone
Contact Company
Company
Contract Number
Karim
: Ibrahim
:+971 56 354 4568
: Alcatel-Lucent
Emirates Telecommunications Corporation
: OXIA 627085
Region
: EMEA
Country
: United Arab Emirates
City
: Abu Dhabi
Site
: ETISALAT ABU DHABI
Product Instance
: 9926 DBS-WCDMA
Logs:
Severity
Priority
: HFB Attached.
: 3
: 3
Product Version
Outage
: LR13.3W
: No
Internal/External:
:Internal
-Best Regards,
Karim Ibrahim
LTE/WCDMA Integration Professional
META WLS Access Network (GNE EMEA)

M(UAE): +
<https://mail.eu.alcatel-lucent.com/owa/redir.aspx?
C=30449991f54743fd85063b63258fd6f2&URL=https%3a%2f%2fmail.eu.alcatellucent.com%2fowa%2fUrlBlockedError.aspx>
971-56-354-4568
M(Egy): +
<https://mail.eu.alcatel-lucent.com/owa/redir.aspx?
C=30449991f54743fd85063b63258fd6f2&URL=https%3a%2f%2fmail.eu.alcatellucent.com%2fowa%2fUrlBlockedError.aspx>
2010-668-44-810
ONNET:
2205 5534
Showing Investigation and Proprietary logs together in time order

separately
Show
Investigation Log and Proprietary Log

1. 06-Jan-2015 10:34
ansikors
Update to Current Summary: Assigned to Workgroup
2. 06-Jan-2015 10:45
ansikors
ALCATEL-LUCENT PROPRIETARY
Outgoing E-mail to Karim (Karim)** CTR ** IBRAHIM ABD EL NABY

Workflow Status: Queued
Subject: Re: Repeated alarm "SYN NOT locked to the Network"/1-5522436
Contact: Karim (Karim)** CTR ** IBRAHIM ABD EL NABY
Agent: ansikors
From: support@alcatel-lucent.com
To: "IBRAHIM ABD EL NABY, Karim (Karim)** CTR **"
<karim.ibrahim@alcatel-lucent.com>
Cc: "BERIDY, AHMED (AHMED)" <Ahmed.Beridy@alcatel-lucent.com>, "KHEDR,
MAHMOUD
(MAHMOUD)" <Mahmoud.Khedr@alcatel-lucent.com>, "ABDEL-HALIM, SAYED (SAYED)"
<Sayed.Abdel-Halim@alcatel-lucent.com>, "ZAHABI, RAMSEY (RAMSEY)"
<ramsey.zahabi@alcatel-lucent.com>, "OKASHA, TAHER (TAHER)"
<Taher.Okasha@alcatel-lucent.com>, "REDA, RAMY (RAMY)"
<ramy.reda@alcatel-lucent.com>, "NABIL GEORGES, MARTIN (MARTIN)** CTR **"
<Martin.Nabil_Georges@alcatel-lucent.com>, "ELHAKIM, Mohamed (Mohamed)** CTR
**" <Mohamed.Elhakim@alcatel-lucent.com>
Bcc:
Sent on:
Dear Customer,
Thank you for contacting the Alcatel-Lucent Welcome Center.

Your request has been processed and the ticket number for your reference is
AR
1-5522436
Please reference that number when contacting Alcatel-Lucent for follow-up.
An Alcatel-Lucent representative will contact you regarding your request.
Kind Regards,
Anna Sikorska
ALCATEL-LUCENT
GLOBAL WELCOME CENTER
Visit OnLine Customer Support for Contact details
3. 07-Jan-2015 01:05
rkosorin
Update to Current Summary: TPS WIP
4. 07-Jan-2015 01:38
rkosorin
Hello Karim,
In the attached hfb there is:

-
The oldest alarm is raised 27.12.2014
It is raised to appr 30 NodeBs
Are those NodeBs Do those NodeBs belong to the same RNC? Do they belong to
the
same network cluster/topology?
The first alarm appeared 27.12.2014 or is the sync related issue older
issue?
Since when is it present? Can you mp any network maintenance to this date?
Please provide network topology design.

Please provide the conclusion of the team, who is doing TP5000 support for
you.
Please choose NodeB where the issue is present and provide output of
following
commands:
/pltf_519/pltf/syn/ info
/pltf_519/pltf/syn/ synh
/pltf/ptp> ptph -a
Login to the NodeB as root and for the vlan used for synchronization please
setup
tcpdump:
xCCM-nodeb-nodeb> su root
Password:nodeb3G
Check your interface config:
xCCM-root-nodeb> route
and vlan config:
xCCM-root-nodeb> ifconfig
And setup the tcpdump for the synchronization carrying vlan:

xCCM-root-nodeb> tcpdump -i your_vlan -vv /tmp/sync_trace.pcap
At the end please run IMT ZAP on the NodeB
Please provide: IMT ZAP, hfb , network snapshot, sync_trace.pcap and the
output of
the commands.
Ticket will be assigned when the initial info & logs are available to me.
BR
Rasto
Update to Current Summary: additional logs & info needed
5. 07-Jan-2015 14:14
rkosorin
Update to Current Summary: logs provided - however still waiting for

complementary
info.
6. 08-Jan-2015 02:01
rories
From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Sent: 07 January 2015 10:47
To: Kosorin, Rastislav (Rastislav)
Cc: ABDEL-HALIM, SAYED (SAYED); OKASHA, TAHER (TAHER); ZAHABI, RAMSEY
(RAMSEY)
Subject: RE: Repeated alarm "SYN NOT locked to the Network"/1-5522436
Hi Rasto,
This issue appeared after any site integration
?
For the HFB I sent I take the 27.12.2014 as reference , @ logs I sent
you all
stability files from 1 Nov.
?
I performed Audit on the HFB I send to you and the most impacted site
is 2513,
so I performed the AP on it.
?
This Nodes are disturbed on 2 the live RNCs (RNC231&RNC235).
All logs are @/EtisalatUAE/1-5522436
The network topology diagram is below:
?
Please we need to find the RC of this issue, a recovery soln & if it
affect
the KPIs or not.
Thanks,
-BR,
Karim
7. 08-Jan-2015 02:01
rories
From: Kosorin, Rastislav (Rastislav)

Sent: Wednesday, January 07, 2015 2:11 PM
To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

(RAMSEY);
RIES, Robert (Robert)
Hello Karim,
Thank you for the info. Ticket is now assigned to Robert, who will support
you.
So the issue is permanent and appeared since integration of the NodeBs,

right?
In your feedback we are still missing the issue view from the perspective of
TP5000.
BR
Rasto
Rastislav KOSORIN
8. 08-Jan-2015 02:02
rories

Sent: Wednesday, January 07, 2015 11:22 AM
To: Kosorin, Rastislav (Rastislav)
(RAMSEY);
RIES, Robert (Robert)
Hi Rasto, Reis,
Yes most of sites are impacted after integration, but not with the same
trend as
per attached, (Audit counted from 27/12/2014).
Im trying to find the who is contact of TP5000 support,
We are waiting your investigation,
Thanks,
-BR,
Karim
9. 08-Jan-2015 02:02
rories
From: RIES, Robert (Robert)

Sent: Wednesday, January 07, 2015 10:59 PM
(RAMSEY)
Hi Karim
Overview:
Based on logs, you sent we can see history of strong synchro break downs of
the
NodeB T_WR_MARRAGE_HALL_MZD_U_2513:
1) IMT has recorded in all flr files progressive streams of alarms like
this:
LOST HIGHER PRIORITY SYN SOURCE

SYN LOCK SOURCE CHANGED
SYN NOT LOCKED TO THE NETWORK
NO GRANT FROM PTP SERVER1
- which are raised and cleared repeatedly every few hours.

All of them are exclusively connected with synchronization corruption.
2) Stability files are full of alarms like this:
For FDD cell

RNC_0021_00020
State change to Disable Failed
RNC_0028_00020
State change to Enable Degraded
RNC_0030_00020
State change to Disable Dependency
RNC_3004_29755
ANO_CELL_NOT_HSDPA_CAPABLE_INFORMATION
RNC_3015_29772
ANO_CELL_NOT_E-DCH_CAPABLE_INFORMATION
State change to Disable Failed
For Control port

RNC_0085_00036
General
RNC_3034_29699
ANO_NODEB_IP_USER_PLANE_FAILURE
All of them are connected with transport quality.

However there is suspicious increase of the size of hfb files the 7th Dec.
This
date (and all subsequent days) we can see very strong increase of all above
mentioned transport alarms.
3) In network snapshot I checked two important parameters connected with PTP

synchro: ptpSyncDuration & ptpAnnounceDuration. Both are set correct to
value 300.
4) I needed to check DSCP values you have set for synch messages. But in tcp
dump
you sent, there are no such synch messages captured, because you captured
only OAM
virtual interface, although for synchronization you have set Telekom
interface:
/pltf_594/emo/ip$ VirtualInterface
========= The Virtual Interface Object Parameters =========

nbVirtualInterface
VirtualInterface
: 2
[0] :
nbTypeOfTraffic
: 1
TypeOfTraffic
: OAM
IpAddress
: 10.235.158.76
VlanId
: 970
VirtualInterface
[1] :
nbTypeOfTraffic
: 3
TypeOfTraffic
: PTP
IpAddress
: 10.241.188.200
VlanId
: 907
/pltf_594/emo/ip$
AP:
Please ask your transport team what changes were performed around the 7th
Dec
either in settings of transport devices, or in general transport topology
especially in the path between TP5000 an NodeB(s).
Perform please tcp dump of telekom interface (VlanId 907). As tcp dump does
not
capture UP traffic, it`s size should not be very big.
And please inspect if you have correctly set next recommended network
settings for
PTP type of synchronization:
Set PTP packets (event & general messages) with highest priority (i.e.
maximum
precedence, typically DSCP CS7) in TP 5000 and in E2E between TP5000 and
NodeBs.
Make sure firewall and IPSec encapsulation are disabled between the TP 5000
servers and NodeB(s).
Configure the dither parameter as enable on the TP 5000.
Best Regards
Robert Ries
10. 09-Jan-2015 14:05

rories

Sent: Friday, January 09, 2015 10:51 AM
(RAMSEY);
EL-MIDANY, AHMED (AHMED)
Hi Karim
Thanks for pcap trace. There is set DSCP value CS7 in PTP synchro message,
which
is correct:
Regarding TP 5000 settings - we are UTRAN support, we have no knowledge

about its
configuration.
Regarding time shift between TP 5000 and WMS time - I thing, it is no
problem from
functionality point of view. But remind that in this case you have not time
correlated alarms, WOs, KPI counters, and so on ... This in result
complicates any
UTRAN investigation.
However when I performed pings from RNC to NodeB and vice versa, the
transport
network showed it`s very bad quality. I set pings to packet size 1500b to
better
see the quality fluctuations. Look at attached document "ping log.docx",
please. I
highlighted lost messages in red in it.
While your transport network is not able to deliver packets without losses,
it has
no sense to investigate any UTRAN issue connected to transport (which is
almost
whichever UTRAN issue).
Best Regards
Robert Ries
11. 09-Jan-2015 14:05

rories
Time Tracking Entry Added - IR01-TSA3
12. 16-Jan-2015 16:38

rories

Sent: Friday, January 09, 2015 12:22 PM
To: RIES, Robert (Robert)
(RAMSEY);
Hi Ries,
Thanks for your feedback,
This is the same result we got from your team regarding the same site and 2
other
which had TX instabillty.
I Agree with it's a clear transport issue and nothing can be done from our
side
but the problem that when we eslcated the issue before to ETC TX team they
said no
issue from thier side, from your point of view what should be the problem
between
our UTRAN and TX( configration mismtsch or BW limitation , or what?) Do you
have
something from the log can give us any indication about why we recieve all
this
packet and the TX is OK?,
Thanks for your usual feedback.
-BR
Karim Ibrahim
13. 16-Jan-2015 16:38

rories
14. 16-Jan-2015 16:40

rories

Sent: Monday, January 12, 2015 4:28 AM
(RAMSEY);
Hi Karim
I cannot investigate issues on transport network as I am responsible only

for
UTRAN issues. Regarding transport I don't know network topology and used
network
devices settings and configurations. BTW I suppose, that even most of them
are not
Alcatel products (however it is not important).
IuB traceroute shows us, that there are more transport devices used (as we
expected):
For download in control plane:

IUBCPLANE => 10.238.14.142 (RNC CP address)
1> ping -src(10.238.14.142) -ip(10.241.188.200) -size(1500) -trace Vr/5 Ip

Icmp
Vr/5 Ip Icmp
IP Trace Route for 10.241.188.200:
Path taken:
Hop 1:
10.238.14.93
(time = 2ms)
Hop 2:
0.0.0.0
(time = 0ms)
Hop 3:
10.241.188.200
(time = 4ms)
ok
2015-01-09 13:06:35.42
For download in user plane:

UPLANE => 10.238.14.158 (RNC UP address)
2> ping -src(10.238.14.158) -ip(10.241.188.200) -size(1500) -trace Vr/5 Ip

Icmp
Vr/5 Ip Icmp
IP Trace Route for 10.241.188.200:
Path taken:
ok
Hop 1:
10.238.14.93
(time = 2ms)
Hop 2:
10.241.188.194
(time = 5ms)
Hop 3:
10.241.188.200
(time = 5ms)
2015-01-09 13:06:46.42
For upload in control plane:
eCCM-nodeb-nodeb> traceroute 10.238.14.142

traceroute to 10.238.14.142 (10.238.14.142), 30 hops max, 38 byte packets
1
10.241.188.194 (10.241.188.194)
10.238.14.94 (10.238.14.94)
10.238.14.93 (10.238.14.93)
For upload in user plane:
0.918 ms
5.269 ms
4.014 ms
0.736 ms
5.256 ms
3.739 ms
0.631 ms
eCCM-nodeb-nodeb> traceroute 10.238.14.158

traceroute to 10.238.14.158 (10.238.14.158), 30 hops max, 38 byte packets
1
10.241.188.194 (10.241.188.194)
10.238.14.94 (10.238.14.94)
10.238.14.93 (10.238.14.93)
0.774 ms
5.317 ms
3.910 ms
0.671 ms
0.722 ms
5.894 ms
3.960 ms
Each of intermediate nodes with above IP addresses has to fulfil conditions

I
wrote in mail with AP in yellow. Transport network must be transparent for
packets
which enter it on the one side and output it on the other side.
Please require transport team to solve any violation of this behavior =>
network
part which are they responsible for must deliver packets without losses !
Without
exception.
Consider, you provided them log with more or less (un)successful pings. And
I
don`t know what they need more ? You (and me also) have no any other tool to
inspect transport network quality. Pings with lost packets are pretty
enough.
Karim, I understand you situation. It is not easy to vindicate UTRAN

disability
while transport team doesn`t want to cooperate on transport issues which
caused
this disability.
However my position is only to point out the problems and if they are caused
by
UTRAN part, also provide solution. I pointed out that UTRAN part has no
possibility to catch synchronization to the network, which is caused based
of
disappearing packets sent as end to end style. So no UTRAN issue. In this

case
provide please permission to close the ticket.
Thank you
Best Regards
Robert Ries
15. 16-Jan-2015 16:40

rories
16. 16-Jan-2015 16:42

rories

Sent: Thursday, January 15, 2015 2:52 PM
(RAMSEY);
EL-MIDANY, AHMED (AHMED); BERIDY, AHMED (AHMED)
Hello Reis,
Kindly regarding the site 2513, we already have another AR which

investigating a
KPIs degradation on the site due to TX instability also, now the TX team
performed
action onsite and the KPIs is stable,
For PTP alarms we still receive PTP alarms but from my mentoring without the
same
rate as before,
Please provide another AP, so you can check from your side.
Waiting your feedback.
Thanks,
-BR,
Karim
17. 16-Jan-2015 16:44

rories

(RAMSEY);
Hi Karim
I performed the same ping check as last time. Finally it is clear of packet
loss.
I checked PTP synch NodeB settings and all seems be OK.
I checked NodeB traffic. There are active sessions in progress.
I downloaded IMT file and no problem I found in it, except Alarms trace:
Filename: /rmem/pltf.Alarms.trace
---------------------------------------------------------TIME REF:
18.0000791630
STAMP REF:
1.0467287408
2015-01-14 01:54:32:474:940588F1 3105.2483390705 [MsgSeq/37b394b0] BTS/0

(BTS EQUIPMENT/0)
RAISED
2015-01-14 01:54:29 NO GRANT FROM PTP SERVER1
2015-01-14 01:54:32:480:940B2647 3105.2483758663 [MsgSeq/37b394b0] BTS
SYNCHRONISATION/0 (BTS EQUIPMENT/0)
CLEARED
2015-01-14 01:54:29 LOST HIGHER PRIORITY SYN SOURCE
2015-01-14 01:54:32:480:940B306E 3105.2483761262 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-14 01:54:29 SYN NOT LOCKED TO THE NETWORK
2015-01-14 01:54:32:480:940B3903 3105.2483763459 [MsgSeq/37b394b0] BTS/0
(BTS EQUIPMENT/0)
CLEARED
2015-01-14 01:59:19:793:C25E04C2 3109.3260941506 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-14 01:59:17 SYN LOCK SOURCE CHANGED
2015-01-14 07:20:16:860:FC888710 3389.4236805904 [MsgSeq/37b394b0] BTS
RAISED
2015-01-14 07:20:16:860:FC889071 3389.4236808305 [MsgSeq/37b394b0] BTS
RAISED
2015-01-14 07:20:16:860:FC889982 3389.4236810626 [MsgSeq/37b394b0] BTS
RAISED
2015-01-14 07:20:16:860:FC88A233 3389.4236812851 [MsgSeq/37b394b0] BTS/0
(BTS EQUIPMENT/0)
RAISED
2015-01-14 07:20:16:880:FC9AD9F9 3389.4238006777 [MsgSeq/37b394b0] BTS

CLEARED
2015-01-14 07:20:16:880:FC9AE446 3389.4238009414 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-14 07:20:16:880:FC9AECF3 3389.4238011635 [MsgSeq/37b394b0] BTS/0
(BTS EQUIPMENT/0)
CLEARED
2015-01-14 07:25:10:208:41565DA6 3394.1096179110 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-14 10:48:38:184:E7976150 3571.3885457744 [MsgSeq/37b394b0] BTS
RAISED
2015-01-14 10:48:38:184:E7976AAE 3571.3885460142 [MsgSeq/37b394b0] BTS
RAISED
2015-01-14 10:48:38:184:E79773D8 3571.3885462488 [MsgSeq/37b394b0] BTS
RAISED
2015-01-14 10:48:38:184:E7977C52 3571.3885464658 [MsgSeq/37b394b0] BTS/0
(BTS EQUIPMENT/0)
RAISED
2015-01-14 10:48:38:193:E7A03141 3571.3886035265 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-14 10:48:38:193:E7A03B8F 3571.3886037903 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-14 10:48:38:193:E7A0442E 3571.3886040110 [MsgSeq/37b394b0] BTS/0
(BTS EQUIPMENT/0)
CLEARED
2015-01-14 10:53:57:015:8B553121 3576.2337616161 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-15 08:40:31:649:56199CF0 4717.1444519152 [MsgSeq/37b394b0] BTS
RAISED
2015-01-15 08:40:31:649:5619A6C9 4717.1444521673 [MsgSeq/37b394b0] BTS
RAISED
2015-01-15 08:40:31:649:5619B03E 4717.1444524094 [MsgSeq/37b394b0] BTS

RAISED
2015-01-15 08:40:31:649:5619B8BC 4717.1444526268 [MsgSeq/37b394b0] BTS/0
(BTS EQUIPMENT/0)
RAISED
2015-01-15 08:40:31:654:561ECB0B 4717.1444858635 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-15 08:40:31:654:561ED4AF 4717.1444861103 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-15 08:40:31:654:561EDD08 4717.1444863240 [MsgSeq/37b394b0] BTS/0
(BTS EQUIPMENT/0)
CLEARED
2015-01-15 08:45:26:434:A0427B03 4721.2688711427 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-15 08:45:32:204:B5C161BD 4721.3049349565 [MsgSeq/37b394b0] BTS
RAISED
2015-01-15 08:45:32:204:B5C16B6C 4721.3049352044 [MsgSeq/37b394b0] BTS
RAISED
2015-01-15 08:45:32:204:B5C174EA 4721.3049354474 [MsgSeq/37b394b0] BTS
RAISED
2015-01-15 08:45:32:204:B5C17D75 4721.3049356661 [MsgSeq/37b394b0] BTS/0
(BTS EQUIPMENT/0)
RAISED
2015-01-15 08:45:32:211:B5C7F0DE 4721.3049779422 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-15 08:45:32:211:B5C7FA72 4721.3049781874 [MsgSeq/37b394b0] BTS
CLEARED
2015-01-15 08:45:32:211:B5C802D2 4721.3049784018 [MsgSeq/37b394b0] BTS/0
(BTS EQUIPMENT/0)
CLEARED
2015-01-15 08:50:49:226:52C16101 4726.1388404993 [MsgSeq/37b394b0] BTS
CLEARED
----------------------------------------------------------
As you can see, the 14th Jan there were three synchro interruptions at
01:54,
07:20, 10:48 (distinguished by different text colors).
The 15th Jan there is just one at 08:40.
Please look at the size of the hfb files from last days. While hfbs up to
the 11th
Jan have many MB, the hfbs from the 14th an 15th Jan are considerably less
than
1MB in size.
The 14th Jan there is just one alarm for NodeB "T_AR_U_2513_MarrageHallMzd"
at
01:28 - "loss of supervision".
The 15th Jan there is no any alarm for this NodeB.
Above described alarms should be caused either by transport fluctuations, or

by hw
failure - eCCM. So if you think, the transport fluctuations are finally
solved,
replace the eCCM board please.
Best Regards
Robert Ries
18. 16-Jan-2015 16:44

rories
19. 23-Jan-2015 18:24

rories

(RAMSEY);
Hello Reis,
Many thanks for your detailed investigation,
I want to mention that the TX action had been taken also on below
sites(Alarm
audit done last week) as we already have KPIs degradation and TX alarms on
them(Also as mentioned before we receiving Sync alarms on them but we took
2513 as
example as it's
stable but
the most impacted site), now the
we still facing Sync alarms.
BTS Name
RNC Name
Count of Alarm Code
M_AR_BaynounhFodJnctn_U_2764 RNC231 88
EP_AR_JebelBarakhEast_U_2525 RNC235 41
ER_AR_BayyaJunction_U_2501
RNC235 32
O_WR_GAYATHI_SOUTH_U2253
RNC235 19
KPIs of most of them
ET_AR_BayaShabiya_U_2375
RNC235 14
EP_AR_AlDhafraSportsAndCultureClub_U_2551
RNC231 10
Should we proceed for CCM change for all them? Or we can find another
corrective
action?, Also please we already receiving the Sync lock alarm all over the
live
sites as you know(see attached the audit done when we opened this AR).
For now we will proceed for CCM change for 2513 as trial and will keep you
updated, Thanks to check other high impacted sites and let us know if you
have any
other corrective action,
Thanks,
-BR,
Karim
20. 23-Jan-2015 18:24

rories
21. 23-Jan-2015 18:24

rories

Sent: Monday, January 19, 2015 5:38 PM

(RAMSEY);
Hello Karim
At the beginning of this ticket investigation we chose one most impacted

NodeB
(2513), we focused on. We found clear RC (the backhaul instability) proved
by
system pings across UTRAN part of network. After some remedial actions on
backhaul
part, you reported essential improvement in KPIs.
Next if you still observe few synch alarms, you can replace CCM as we talked
about
already.
However if you observe similar symptoms even on another sites, these issues
should
be investigated in another ARs. I keep our general rule => one ticket = one
issue.
So for another NodeBs, open new tickets, please.
But look at the attached log. I performed at least basic pings also for site
2764,
even though I shouldn`t.
There are clear visible the same backhaul defects caused loosing of ping
commands
as in case of site 2513.
As a result - it has no sense to change CCMs on all other NodeBs, while

there is
such unreliable transport network. First manage solving of the backhaul
problems,
please.
Best Regards
Robert Ries
22. 23-Jan-2015 18:24

rories

Sent: Tuesday, January 20, 2015 11:07 PM
(RAMSEY);
Hi Reis,
Thanks for your feedback, we will try to change the CCM and see the if there
are
any enhancement of the Sync alarms or not.
For the Rule we respect it for sure and following for all tickets, but this
ticket
opened to include all live sites on the network which have the same Sync
alarms..
so when we solve the issue for some sites or one sites from my point of view
not
mean to close the ticket and open another one with the same description.
As you know we still investigation with the transport team and will keep you
updated.
Thanks for your usual support,
Thanks,
-BR,
Karim
23. 23-Jan-2015 18:25

rories

Sent: Thursday, January 22, 2015 9:22 AM
(RAMSEY);
Hi Reis,
Kindly note that we changed the CCM for the site 2513 @ 17.1.2015 but the
site
still suffering from PTP SYNC alarms as attached, as you know the site is
stable
from TX's side.
Please can you check and propose a soln for this issue.
Thanks,
-BR,
Karim
24. 23-Jan-2015 18:25

rories

Sent: Thursday, January 22, 2015 2:34 PM
(RAMSEY);
Hi Karim
Look at attached xls file, please. I added the new one sheet "Alarm
duration"
where are just exported columns H (Event Time) and L (Clear Time) from the
sheet
"HFB_2513". I highlighted in red those time values where there was any
observable
time shift present between these two time values (in any words, where there
is
some difference between Event and Clear Time). But alarms, which occurred
and
disappeared in the same second are in black. So in this sheet you can see
duration
of alarms from site 2513 more clearly.
In summary there is 87 alarms which occurred and disappeared just in the
same
second and only 36 alarms which shows any time duration. And as you can see
this
duration is in average only few minutes.
So what results from this extrapolation ? NodeB 2513 has no systematic

problem
with synchronization. If it would had, the alarms would be systematically
present.
But we can see only occasional synchro fluctuations which last from less
than one
second up to just few minutes in fact.
Moreover column Q in the sheet "HFB_2513" shows you Probable cause of

incoming
alarms. Here you can see, only two type of records "lossOfSignal" and
"communicationsProtocolError" which points to transport network quality
again.
As issue was answered and if there is nothing else I can help you regarding
this
issue, kindly could I close the ticket ?
Best Regards
Robert Ries
25. 23-Jan-2015 18:25

rories

Sent: Friday, January 23, 2015 10:44 AM
(RAMSEY);
Hello Reis,
Thanks for this detailed investigation, I have one question then we can
close this
AR, as you mentioned below regarding column Q, I can see also that the
Specific
Problem is "BTS/0 NO GRANT MESSAGE FROM PTP SERVER 1"
My question is: Based on your investigation, NO issue with our PTP server?
And
it's totally TX issue? Or may we have any configuration mismatch or PTP
failure
affecting this issue?
Probable Cause
Specific Problem
communicationsProtocolError BTS/0 NO GRANT MESSAGE FROM PTP SERVER 1
Thanks,
-BR,
Karim
26. 23-Jan-2015 18:25

rories

(RAMSEY);
Hello Karim
Look to the alarm sequences in the xls file HFB_2513 again, please. These
alarm
chunks exhibits a cyclic repetition in similar sequences like this (with
comments
copied from Alarm Reference Guide):
=================================================
BTS_0358_00002 - LOST HIGHER PRIORITY SYN SOURCE => This alarm indicates
that the
NodeB is not able to be synchronized.
BTS_0007_00002 - SYN NOT LOCKED TO THE NETWORK => The alarm indicates that
the BTS
is unable to derive reference timing from the Pulse Code Modulation (PCM) or
from
the synchronization sources.
BTS_0388_00033 - NO GRANT MESSAGE FROM PTP SERVER 1 => This alarm indicates
that
preferred PTP time server does not provide GRANT message to the NodeB
REQUEST
UNICAST TRANSPORTATION messages.
BTS_0333_00002 - SYN LOCK SOURCE CHANGE => This alarm indicates that SYN
LOCK
Source of a BTS is changed.
================================================
Important fact in this case is, that each chunk starts with alarm
BTS_0358_00002
with indicates synchronization lost (probable cause "lossOfSignal"). And
alarm
BTS_0388_00033 is just consequence of foregoing alarms indicated lost of
synchronization to the network.
In case you have any configuration mismatch in your PTP server, the NodeBs
synchronization problems would be permanent.
As issue was answered and if there is nothing else I can help you regarding
this
issue, kindly could I close the ticket ?
Best Regards
Robert Ries
27. 23-Jan-2015 18:27

rories

(RAMSEY);
Hi Reis,
Thanks again, You can close the AR,
Thanks,
-BR,
Karim
Update to Current Summary: Issue confirmed as backhaul instability.
28. 23-Jan-2015 18:29

rories
Update to Current Summary: Closure approval granted

Resolution
Issue confirmed as backhaul instability.

End of Assistance Request 1-5522436
New AR:
View AR | AR without Proprietary | OLCS

History | States | Assignment | Timetracking
AR Text Search | More | CARES Home

Assistance Request 1-5522436: View AR

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Assistance Request 1-5522436: View AR

Uploaded by

Copyright:

Available Formats

New AR:

View AR | AR without Proprietary

Assistance Request 1-5522436

TEC support needed to check the RC of this issue, and if

affecting the KPIs or not, the network deployed with PTP

Closure approval granted

Closed : Solution/Service Provided

Service Request Remote Technical Support

No Change Request is linked to this AR

ETISALAT ABU DHABI

9926DBS WCDMA-Etisalat : In Service

Emirates Telecommunications Corporation

United Arab Emirates

Karim IBRAHIM ABD EL NABY

Alcatel-Lucent : Open on behalf of Customer

Targets for Restore & Resolve were extended by 16 days

242895 (OXIA 627085)

Covered Service TS 24x7 (Gold, Wireless Mobile Access)

This request is entitled to service.

From: "IBRAHIM ABD EL NABY, Karim (Karim)** CTR **"

"SYN NOT locked to the Network"

Repeated PTP alarm fail over the network,

on behalf of the alarm SYN NOT locked to the Network

: TEC support needed to check the RC of this

Contact Given Name

:+971 56 354 4568

Emirates Telecommunications Corporation

: United Arab Emirates

: ETISALAT ABU DHABI

META WLS Access Network (GNE EMEA)

Showing Investigation and Proprietary logs together in time order

Investigation Log and Proprietary Log

Update to Current Summary: Assigned to Workgroup

Outgoing E-mail to Karim (Karim)** CTR ** IBRAHIM ABD EL NABY

Thank you for contacting the Alcatel-Lucent Welcome Center.

GLOBAL WELCOME CENTER

Visit OnLine Customer Support for Contact details

Update to Current Summary: TPS WIP

In the attached hfb there is:

The oldest alarm is raised 27.12.2014

It is raised to appr 30 NodeBs

Please provide network topology design.

And setup the tcpdump for the synchronization carrying vlan:

Update to Current Summary: logs provided - however still waiting for

From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Subject: RE: Repeated alarm "SYN NOT locked to the Network"/1-5522436

This issue appeared after any site integration

This Nodes are disturbed on 2 the live RNCs (RNC231&RNC235).

All logs are @/EtisalatUAE/1-5522436

The network topology diagram is below:

From: Kosorin, Rastislav (Rastislav)

Cc: ABDEL-HALIM, SAYED (SAYED); OKASHA, TAHER (TAHER); ZAHABI, RAMSEY

So the issue is permanent and appeared since integration of the NodeBs,

From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Im trying to find the who is contact of TP5000 support,

We are waiting your investigation,

From: RIES, Robert (Robert)

LOST HIGHER PRIORITY SYN SOURCE

- which are raised and cleared repeatedly every few hours.

2) Stability files are full of alarms like this:

For FDD cell

State change to Disable Failed

State change to Enable Degraded

From: "IBRAHIM ABD EL NABY, Karim (Karim) CTR "

Outgoing E-mail to Karim (Karim) CTR IBRAHIM ABD EL NABY

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

To: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR

From: IBRAHIM ABD EL NABY, Karim (Karim) CTR