MTP2 Link went down & came up in idle state in MSC.

MTP2 link went down & came up in idle state.

Steps:
1) Single MTP2 link with linkset id:1, link id:0 was configured at MSC side & wa
s up & running.
2) Around 10:38 pm (24/11/2011), link went down & then came up. (Observed from B
SC alarms)
Observations:
==============
1) Uptime of both CP & PP were found different.
2) From HA_PP logs & HA_CP logs, "PLATFORM IS NOT HEALTHY" was observed.
MSC : 5.3.13.002 + patches
BSC : 2.6.1.001
OMC : 12.0.0.001

From HA_CP logs :

24 170814:579 D[ 0][HA][broadcast_healthcheck][DEBUG]::telecomCnt[36] connectio
n_entries[55] gCPLDState[73]
24 170814:586 D[ 0][HA][sendHealthToPp][DEBUG]::Enter the program
24 170814:587 D[ 0][HA][broadcast_healthcheck][DEBUG]::******* CARD HEALTH **
******* [HEALTHY]
24 171041:658 D[ 0][HA][main][DEBUG]::***** HA STARTED *****
24 171041:659 D[ 0][HA][main][DEBUG]::Health Interval[1000] millisecs Max KA[4]
24 171041:659 D[ 0][HA][updateHealthParams][DEBUG]::healthInterval 1000 max num
health msg 4
24 171041:659 D[ 0][HA][updateHealthParams][DEBUG]::App health interval 1000 mi
lisecs
24 171041:659 D[ 0][HA][updateHealthParams][DEBUG]::App health max ka 4
24 171041:659 D[ 0][platformLib][checkProcessor][DEBUG]::Processor matches [VNL
-ICC-CP]
24 171041:659 D[ 0][HA][haInit][DEBUG]::Enter the program
24 171041:677 D[ 0][platformLib][readMcReg][DEBUG]::read(0x0005) = 0x03
24 171041:694 D[ 0][platformLib][readMcReg][DEBUG]::read(0x0049) = 0x05
24 171041:722 D[ 0][platformLib][readMcReg][DEBUG]::read(0x004a) = 0x03
24 171041:739 D[ 0][platformLib][readMcReg][DEBUG]::read(0x004b) = 0x0d
24 171041:740 D[ 0][MPILIB][isMpiMpiConnected][DEBUG]::PEER PROC MPI sockfd: 8
24 171041:740 D[ 0][MPILIB][isMpiMpiConnected][DEBUG]::PEER PROC MPI urgent soc
kfd: 13
24 171041:740 D[ 0][MPILIB][isMpiMpiConnected][DEBUG]::MPI-MPI are connected, c
ontinuing with the registration
24 171041:740 D[ 0][MPILIB][mpi_register][DEBUG]::IP read: 192.168.6.127
24 171041:740 D[ 0][MPILIB][mpi_register][DEBUG]::getsockopt_SB, ret: 65536
24 171041:740 D[ 0][MPILIB][mpi_register][DEBUG]::getsockopt_SB after set, ret:
65536
24 171041:741 D[ 0][MPILIB][mpi_register][DEBUG]::getsockopt_SB, ret: 65536
24 171041:741 D[ 0][MPILIB][mpi_register][DEBUG]::getsockopt_SB after set, ret:
65536
24 171041:741 D[ 0][MPILIB][mpi_register][DEBUG]::calling select on fds: 6 7
24 171041:742 D[ 0][MPILIB][mpi_register][DEBUG]::Line 765
24 171041:742 D[ 0][MPILIB][mpi_register][DEBUG]::Received msg from MPI for: HA

-CP opcode: 241

24 171041:742 D[ 0][MPILIB][mpi_register][DEBUG]:: HA-CP registered with MPI
for NORMAL prio
24 171041:742 D[ 0][MPILIB][mpi_register][DEBUG]:: HA-CP registered with MPI
for URGENT prio
24 171041:742 D[ 0][HA][createHaAppsList][DEBUG]::Enter the program
24 171041:742 D[ 0][platformLib][SemaphoreWait][DEBUG]::Semaphore released retu
rn val [0] returning success
24 171041:742 D[ 0][HA][createHaAppsList][DEBUG]::VERSION from SHM ::0.0.0
24 171041:742 D[ 0][HA][createHaAppsList][DEBUG]:: gNumPlatformApp[4] gHaAppsLi
st[268579232] In App List telecomCnt[36], gMCnt[36], gSmCnt[0], gNmCnt[0]
24 171041:742 D[ 0][HA][createHaAppsList][DEBUG]::Now DETACHING Shared Memory
24 171041:743 D[ 0][HA][createAISexecThread][DEBUG]::Enter the program
24 171041:743 D[ 0][HA][createAppTxThread][DEBUG]::Enter the program
24 171041:743 D[ 0][HA][createToggleHaMonitorThread][DEBUG]::enter the program
24 171041:743 D[ 0][HA][AISexecFunc][DEBUG]::AISExec entered
24 171041:744 D[ 0][HA][AISexecFunc][DEBUG]::libais_server_fd: 10.
24 171041:744 D[ 0][HA][AISexecFunc][DEBUG]::max_fd: 10.
24 171041:744 D[ 0][HA][releasePiSem][DEBUG]::value = 1
24 171041:744 D[ 0][HA][appTx][DEBUG]::MQ attr size =1048
24 171041:755 D[ 0][HA][main][DEBUG]::HA Init success
24 171041:965 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::Appl
ication [PM][118]found in the haApplicationsL
ist
24 171041:965 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::AIS
Executive: Application [PM] registered with f
d [14]
24 171041:965 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::BPP
COUNT[1] TELECOM COUNT[0] gNumTelecomApp[36]
24 171041:965 D[ 0][HA][message_handler_req_amf_ComponentConfigure][DEBUG]::COM
PONENT CONFIGURE received from [PM] appID[118
]
24 171042:720 D[ 0][HA][handleHaHappMessage][DEBUG]::Enter the program
24 171042:772 D[ 0][HA][broadcast_healthcheck][DEBUG]::telecomCnt[0] connection
_entries[15] gCPLDState[0]
24 171042:773 D[ 0][HA][calculateOverAllHealth][DEBUG]::PLATFORM IS NOT HEALTHY
24 171042:773 D[ 0][HA][sendHealthToPp][DEBUG]::Enter the program
24 171042:774 D[ 0][HA][broadcast_healthcheck][DEBUG]::******* CARD HEALTH **
******* [SICK]
24 171042:882 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::Appl
ication [MPI][123]found in the haApplications
List
24 171042:882 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::AIS
Executive: Application [MPI] registered with
fd [15]
24 171042:882 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::****
** "MPI" is in Disable state ******
24 171042:882 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::BPP
COUNT[1] TELECOM COUNT[0] gNumTelecomApp[36]
24 171042:882 D[ 0][HA][message_handler_req_amf_ComponentConfigure][DEBUG]::COM
PONENT CONFIGURE received from [MPI] appID[12
3]
24 171042:894 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::Appl
ication [UA][116]found in the haApplicationsL
ist
24 171042:895 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::AIS
Executive: Application [UA] registered with f
d [16]
24 171042:895 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::BPP
COUNT[2] TELECOM COUNT[0] gNumTelecomApp[36]

24 171042:895 D[ 0][HA][message_handler_req_amf_ComponentConfigure][DEBUG]::COM
PONENT CONFIGURE received from [UA] appID[116
]
24 171042:930 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::Appl
ication [RMON][114]found in the haApplication
sList
24 171042:930 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::AIS
Executive: Application [RMON] registered with
fd [17]
24 171042:930 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::BPP
COUNT[3] TELECOM COUNT[0] gNumTelecomApp[36]

From HA_PP logs :

24 170813:587 D[ 0][HA][handleHaHappMessage][DEBUG]::Enter the program
24 170814:493 D[ 0][HA][broadcast_healthcheck][DEBUG]::telecomCnt[3] connection
_entries[22] gCPLDState[0]
24 170814:493 D[ 0][HA][sendHealthToCp][DEBUG]::enter the program
24 170814:495 D[ 0][HA][broadcast_healthcheck][DEBUG]::******* PP HEALTH ****
***** [HEALTHY]
24 170814:587 D[ 0][HA][handleHaHappMessage][DEBUG]::Enter the program
24 171040:665 D[ 0][HA][main][DEBUG]::***** HA STARTED *****
24 171040:665 D[ 0][HA][main][DEBUG]::Health Interval[1000] millisecs Max KA[4]
24 171040:665 D[ 0][HA][updateHealthParams][DEBUG]::healthInterval 1000 max num
health msg 4
24 171040:665 D[ 0][HA][updateHealthParams][DEBUG]::App health interval 1000 mi
lisecs
24 171040:665 D[ 0][HA][updateHealthParams][DEBUG]::App health max ka 4
24 171040:666 D[ 0][platformLib][checkProcessor][DEBUG]::Processor matches [VNL
-ICC-PP]
24 171040:666 D[ 0][HA][haInit][DEBUG]::Enter the program
24 171040:668 D[ 0][platformLib][readMcReg][DEBUG]::read(0x0005) = 0x03
24 171040:673 D[ 0][platformLib][readMcReg][DEBUG]::read(0x0049) = 0x05
24 171040:675 D[ 0][platformLib][readMcReg][DEBUG]::read(0x004a) = 0x03
24 171040:677 D[ 0][platformLib][readMcReg][DEBUG]::read(0x004b) = 0x0d
24 171040:677 D[ 0][MPILIB][isMpiMpiConnected][DEBUG]::PEER PROC MPI sockfd: 8
24 171040:677 D[ 0][MPILIB][isMpiMpiConnected][DEBUG]::PEER PROC MPI urgent soc
kfd: 19
24 171040:678 D[ 0][MPILIB][isMpiMpiConnected][DEBUG]::MPI-MPI are connected, c
ontinuing with the registration
24 171040:678 D[ 0][MPILIB][mpi_register][DEBUG]::IP read: 192.168.6.128
24 171040:678 D[ 0][MPILIB][mpi_register][DEBUG]::getsockopt_SB, ret: 65536
24 171040:678 D[ 0][MPILIB][mpi_register][DEBUG]::getsockopt_SB after set, ret:
65536
24 171040:679 D[ 0][MPILIB][mpi_register][DEBUG]::getsockopt_SB, ret: 65536
24 171040:679 D[ 0][MPILIB][mpi_register][DEBUG]::getsockopt_SB after set, ret:
65536
24 171040:679 D[ 0][MPILIB][mpi_register][DEBUG]::calling select on fds: 6 7
24 171040:680 D[ 0][MPILIB][mpi_register][DEBUG]::Line 765
24 171040:680 D[ 0][MPILIB][mpi_register][DEBUG]::Received msg from MPI for: HA
-PP opcode: 241
24 171040:680 D[ 0][MPILIB][mpi_register][DEBUG]:: HA-PP registered with MPI
for NORMAL prio
24 171040:680 D[ 0][MPILIB][mpi_register][DEBUG]:: HA-PP registered with MPI
for URGENT prio
24 171040:680 D[ 0][HA][createHaAppsList][DEBUG]::Enter the program

24 171040:680 D[ 0][platformLib][SemaphoreWait][DEBUG]::Semaphore released retu

rn val [0] returning success
24 171040:680 D[ 0][HA][createHaAppsList][DEBUG]::VERSION from SHM ::0.0.0
24 171040:680 D[ 0][HA][createHaAppsList][DEBUG]:: gNumPlatformApp[4] gHaAppsLi
st[268575136] In App List telecomCnt[3], gMCn
t[3], gSmCnt[0], gNmCnt[0]
24 171040:680 D[ 0][HA][createHaAppsList][DEBUG]::Now DETACHING Shared Memory
24 171040:680 D[ 0][HA][createAISexecThread][DEBUG]::Enter the program
24 171040:681 D[ 0][HA][createAppTxThread][DEBUG]::Enter the program
24 171040:681 D[ 0][HA][createToggleHaMonitorThread][DEBUG]::enter the program
24 171040:681 D[ 0][HA][main][DEBUG]::HA Init success
24 171040:681 D[ 0][HA][AISexecFunc][DEBUG]::AISExec entered
24 171040:682 D[ 0][HA][AISexecFunc][DEBUG]::libais_server_fd: 10.
24 171040:682 D[ 0][HA][AISexecFunc][DEBUG]::max_fd: 10.
24 171040:683 D[ 0][HA][releasePiSem][DEBUG]::value = 1
24 171040:683 D[ 0][HA][appTx][DEBUG]::MQ attr size =1048
24 171041:040 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::Appl
ication [PM][119]found in the haApplicationsL
ist
24 171041:040 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::AIS
Executive: Application [PM] registered with f
d [14]
24 171041:040 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::BPP
COUNT[1] TELECOM COUNT[0] gNumTelecomApp[3]
24 171041:040 D[ 0][HA][message_handler_req_amf_ComponentConfigure][DEBUG]::COM
PONENT CONFIGURE received from [PM] appID[119
]
24 171041:691 D[ 0][HA][broadcast_healthcheck][DEBUG]::telecomCnt[0] connection
_entries[15] gCPLDState[0]
24 171041:695 D[ 0][HA][calculateOverAllHealth][DEBUG]::PLATFORM IS NOT HEALTHY
24 171041:695 D[ 0][HA][sendHealthToCp][DEBUG]::enter the program
24 171041:695 D[ 0][HA][sendHealthToCp][DEBUG]::enter the program
24 171041:696 D[ 0][HA][broadcast_healthcheck][DEBUG]::******* PP HEALTH ****
***** [SICK]
24 171041:750 D[ 0][HA][handleHaHappMessage][DEBUG]::Enter the program
24 171041:828 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::Appl
ication [UA][117]found in the haApplicationsList
24 171041:828 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::AIS
Executive: Application [UA] registered with fd [15]
24 171041:828 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::BPP
COUNT[2] TELECOM COUNT[0] gNumTelecomApp[3]
24 171041:828 D[ 0][HA][message_handler_req_amf_ComponentConfigure][DEBUG]::COM
PONENT CONFIGURE received from [UA] appID[117]
24 171042:137 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::Appl
ication [RMON][115]found in the haApplicationsList
24 171042:137 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::AIS
Executive: Application [RMON] registered with fd [16]
24 171042:137 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::BPP
COUNT[3] TELECOM COUNT[0] gNumTelecomApp[3]
24 171042:138 D[ 0][HA][message_handler_req_amf_ComponentConfigure][DEBUG]::COM
PONENT CONFIGURE received from [RMON] appID[115]
24 171042:182 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::Appl
ication [PI][111]found in the haApplicationsList
24 171042:182 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::AIS
Executive: Application [PI] registered with fd [17]
24 171042:182 D[ 0][HA][message_handler_req_amf_componentregister][DEBUG]::BPP
COUNT[4] TELECOM COUNT[0] gNumTelecomApp[3]
24 171042:182 D[ 0][HA][message_handler_req_amf_ComponentConfigure][DEBUG]::COM
PONENT CONFIGURE received from [PI] appID[111]

Issue
MSC :
BSC :
OMC :

observed with the following release :

5.3.13.002 + patches
2.6.1.001
12.0.0.001

MSC_PRC_LOGS & var logs of MSC CP & PP kept at:

192.168.4.236:/mnt/msc/R2.6mantis/19411