1.0 DDB PHASE 1....................................................................................................... 2 1.1 INTRODUCTION.................................................................................................. 2 1.2 DEFINITIONS....................................................................................................... 2 1.2.1 DDB Quiet Period......................................................................................... 2 1.2.

2 OAM DDB audit............................................................................................ 3 1.2.3 DDB real-time checksums............................................................................3 1.2.4 New DDB statistics....................................................................................... 3 1.2.5 DDB wild write audit.................................................................................... 4 1.2.6 DDB Timers.................................................................................................. 4 1.3 DDB debugging.................................................................................................. 5 1.3.1 Description of DDB send-msg debug commands table................................5 1.3.2 dbg-ddb command....................................................................................... 6 1.3.3 send-msg to dbg-ddb command mapping...................................................7 2.0 DDB PHASE 2........................................................................................................ 9 2.1 Mechanism DIFFERENCES WITH DDB phase 1...................................................9 2.2 DDB wild write audit.......................................................................................... 9 2.3 NEW DDB Timers............................................................................................. 10 3.0 METHODS OF RESOLUTION OF DDB ISSUES ......................................................11 3.1 Adding a secondary route by using a dummy route to trigger a route state change .................................................................................................................. 11 3.2 CARDS report DDB MISMATCH checksum FOR LINKSET table .......................13 3.3 CARDS report DDB MISMATCH checksum for LInk table ..............................14

1.0 DDB PHASE 1

1.1 INTRODUCTION The accuracy of the DDB checksums is determined based on the DDB idle period collected from the MTP cards. If all idle periods reported are greater than the DDB quiet period, the checksums are deemed to be accurate. The consistency of the DDB is determined based on the cross-reference of all checksums on all cards for mistmatch. The evaluation of the DDB consistency is only achieved if the DDB checksums are deemed accurate. The DDB audit is automatic and periodic. The user may also manually trigger an audit. The user may also use DDBAUDTIMER SS7OPTS option to enable and disable the background audit. If any inconsistencies are detected by the OAM as a result of a background audit or manual audit, a system alarm is raised in-order for the user to take the appropriate steps to remedy the DDB corruption. The alarm is cleared after an audit with no inconsistencies (either background or manual).

The MTP cards (LIM and SCCP), calculate and record DDB checksums in real-time as DDB updates are applied. When an audit request is received from the OAM, each MTP card collect all of its DDB checksums and send them in a reply to the OAM card. In addition to providing the OAM with the DDB checksums, the MTP cards, would also attach a time stamp representing the time elapsed in milliseconds since the last DDB update was processed (DDB idle period). The MTP cards implement new DDB statistics. The statistics measure the DDB update rates. Their intent is to help in quantifying the DDB update traffic. The new DDB statistics are not EAGLE measurements and are not saved when a card reboots. 1.2 DEFINITIONS 1.2.1 DDB Quiet Period The DDB quiet period is defined as the minimum DDB idle time (no DDB updates applied) after which it is safe to assume that all DDB updates in the system have been processed and no outstanding in-flight multi-cast update exists. The DDB quiet period shall be greater than the sum of: 1. The maximum time it takes a multicast to reach all cards 2

2. The maximum time for a DDB multicast to be processed by the receiving card when there are no other updates to process. 3. The maximum time for a broadcast to be received and processed by a card. The addition of the broadcast time is due to the fact that the DDB audit uses IMT broadcast to collect checksum data. Since the broadcast maybe sent on both A&B IMT buses, a card may receive duplicate audit requests while another card may miss one of the two. The quiet period as defined above insures that if a card replies to any of the two broadcasts it receives, the data reported stays valid (see timing diagram below).

1.2.2 OAM DDB audit The DDB audit mechanism is triggered either: 1. Manually, using a aud-data command (refer to section 3.4.2) 2. Periodically by a background process. The audit process consists of an audit message broadcast [1] (using IMT broadcast capability) by the active OAM card to all cards in the system. Upon reception of the audit request, each MTP card in the system, would read the real-time DDB checksums for each table and save them into a reply message. Non MTP cards in the system shall ignore the DDB audit message and shall not output any error message. 1.2.3 DDB real-time checksums Currently, the DDB checksums are calculated for each DDB table when requested by the user (OAM card). The DDB checksum for each table is a calculated as a sum of each entry checksum. Each entry checksum is a Polynomial checksum. The current computation of DDB checksums is time consuming. The DDB checksums shall be updated in real-time as follow: when a DDB entry is updated, the polynomial checksum is re-calculated and the overall DDB table checksum is adjusted accordingly based on the old and new entry checksums. All DDB tables shall be expanded to include the entry DDB checksum. 1.2.4 New DDB statistics New DDB statistics are proposed for implementation on MTP cards. These are standalone statistics and are not measurements. These statistics are implemented on the MTP cards as counters saved in the memory. They are reported with each DDB audit reply and on demand via the dbgddb command (see section 1.3.2).
[1]

True group broadcast feature is expected to be implemented in the future when all MTP cards will become a part of the MTP broadcast list. When this feature becomes available, individual cards and GPLs will not need to filter and discard broadcast messages they dont need. Also, the group broadcast feature will have the COM processor send only one copy of the message to the APPL, hence obviating the need to discard duplicates in the future

The statistics are saved on the OAM with a time stamp when reported. Even though the checksums in the audit replies maybe discarded due to a short DDB idle period, the DDB statistics are valid and shall be saved in memory. The following DDB statistics shall be implemented: 1. The number of DDB updates received and applied. These counters shall be synchronized during the DDL phase. 2. The average DDB updates per 100 ms. 3. The maximum DDB updates per 100 ms. 4. The average DDB idle period. 5. The maximum and minimum DDB idle period.

1.2.5 DDB wild write audit Since the DDB checksums are updated in real-time, a wild write to any DDB table would result in the DDB checksum(s) to be out of sync. A wild write may be caused by: 1. A DDB update which does not update the checksums 2. A memory violation caused by a software error An audit process is implemented on all MTP cards. The process is periodic and time sliced. When triggered the processes runs on a 5 ms time-slice. Every time-slice the process computes a maximum number of DDB entry checksums. 1.2.6 DDB Timers Description Time interval between consecutive DDB audits Value(s) 5 to 1440 mins Default: 10 mins 2, 5, 10 secs 2000 ms 60 mins 5 ms

Timer DDB background audit timer

DDB audit retry timer DDB audit replies timer DDB wild write audit timer DDB wild write time-slice

Time between retries when DDB idle period threshold is not met. There shall be only 3 retries. Time within which all audit replies are received. Timer interval between consecutive Wild write audits Time intervals between each time-slice of DDB wild write audit process

DDB Quiet period

Default value for DDB quiet period

500 ms

1.3 DDB DEBUGGING Currently, EAGLE provides the user with a multitude of send-msg commands that may help in debugging the dynamic database. The existing debug functionality offers the user commands to access individual DDB link, link-set and route entries. In addition the user is provided with commands to audit DDB at a debug level as follow: Per card DDB table audit: For a given source card and a reference card, the source card sends an audit request for the specified table. The reference card responds with the checksum for the table. Per entry DDB table audit: If the source cards in the above audit (per card audit) detect a table checksum mismatch, it audits every entry in the table with the reference card. This process starts at entry index zero. The source card sends a per entry audit request to the reference card with the table ID and starting index. The reference card calculates as many entry checksums as it can pack in a reply.

The existing debug send-msg commands are consolidated into one dbg-ddb command. The new dbg-ddb command provides an easy to use wrapper. For example, in order to display a DDB route or link entry, the user is required to provide the entry index if the send-msg command is used. The new dbg-ddb command allows the user to avoid the time consuming task of figuring out entry indexes and simply provide the link location or route DPC to display a specific entry. The table below describes the send message commands consolidated by the new dbg-ddb command. 1.3.1 Description of DDB send-msg debug commands table Dest. Functi Description Appl. on He1 H83 This command is used to display an entry within a cards DDB tables. The DDB tables supported are: Link, Link-set and Route Parameters: He1 H86 D0 : DDB table (0:Link, 1:Link-set, 2: Route) D1 : Least significant byte of the table entry index D2: Most significant byte of the table entry index

Card to System DDB table audit. (Unicast). This command audits a specified DDB table. A per table audit message is issued to a selected reference MTP 5

card. The card selection is based on an internal list. Every time this command is invoked the next card in the list is selected. If the process of the per table audit reply leads to no checksum mismatches, a new per table audit is sent to the next card in the list. If a mismatch is detected, a per entry audit process is started with the reference card. When this process ends, the total number of mismatches and the first entry mismatch are displayed. Parameters: He1 H85 D0 : DDB table (0:Link, 1:Link-set, 2: Route) Card to System DDB table audit. (Broadcast) This command audits a specified DDB table. per table audit messages are sent to every MTP card in the system. The messages are sent 10 at once every 20ms. The per table audit replies are processed and the checksums for the reference cards are stored. When all per table audit replies are received, processing is started. If a table checksum mismatch is found, a per entry audit is started with the reference card that reported the mismatch. Parameters: He1 H80 D0 : DDB table (0:Link, 1:Link-set, 2: Route) Card to card table audit. This command does a per entry audit for the specified table with the reference card specified. Parameters: D0 : DDB table (0:Link, 1:Link-set, 2: Route) D1 : Reference card IMT address

1.3.2 dbg-ddb command This is a new command used to debug DDB. Parameter LOC TBL M/O M M Description Card location: the location of the card where the command would be sent. DDB table: Values: lnk, ls, rte Note: Only Links, Link-sets and route DDB tables are

supported. There is no support for the remaining DDB tables. ACTION M disp: Displays a DDB table entry stats: Sends a message to the card to display the DDB statistics. audit: Audits a specific DDB table Audit type: Using unicast or multicast Values: MC, UC Default: MC DDB table index: optional. Used only when action=disp Mutually exclusive with options: LSN, RLOC,LINK and DPC Link-set name. Used only when action=disp and tbl=ls Mutually exclusive with options: TIDX, RLOC,LINK and DPC Reference card location. Used only when action=disp and tbl=lnk or when action=AUD and audtype=UC Mutually exclusive with options: TIDX ,LSN and DPC Link (A..B31) Used only when action=disp and tbl=lnk Mutually exclusive with options: TIDX , LSN and DPC Destination point code. Used only when action=disp and tbl=rte Mutually exclusive with options: TIDX,LSN, RLOC and LINK The following table shows the translation between the dbg-ddb command and the currently implemented send-msg commands. 1.3.3 send-msg to dbg-ddb command mapping

AUDTYPE

TIDX LSN

O O

RLOC

LINK

DPC

DBG-DDB command Tbl=LNK:action=DISP:TIDX=n Tbl=LNK:action=DISP:RLOC=x:LINK=y The RLOC and LINK shall be converted into a link index (n). Tbl=LS:action=DISP:TIDX=n Tbl=LS:action=DISP:LSN=<link-set name> The link-set name shall be converted into a
1 2

DS 1 1 1 1

DA He1 He1 He1 He1

F H8 3 H8 3 H8 3 H8 3

D0 0 0 1 1

D1 nLO1 nLO nLO nLO

D2 nHI2 nHI nHI nHI

link-set index (n). Tbl=RTE:action=DISP:TIDX=n Tbl=RTE:action=DISP:DPC=a-b-c The DPC shall be converted into a route index (n) Tbl=LNK:action=AUD:audtype=MC Tbl=LS:action=AUD:audtype=MC Tbl=RTE:action=AUD:audtype=MC Tbl=LNK:action=AUD:audtype=UC Tbl=LS:action=AUD:audtype=UC Tbl=RTE:action=AUD:audtype=UC Tbl=LNK:action=AUD:audtype=UC:RLOC=x The alternative location shall be translated into an IMT address (A) Tbl=LS:action=AUD:audtype=UC:RLOC=x The alternative location shall be translated into an IMT address (A) Tbl=RTE:action=AUD:audtype=UC:RLOC=x The alternative location shall be translated into an IMT address (A)

1 1 1 1 1 1 1 1 1 1 1

He1 He1 He1 He1 He1 He1 He1 He1 He1 He1 He1

H8 3 H8 3 H8 5 H8 5 H8 5 H8 6 H8 6 H8 6 H8 0 H8 0 H8 0

2 2 0 1 2 0 1 2 0 1 2

nLO nLO

nHI nHI

A A A

2.0 DDB PHASE 2

2.1 MECHANISM DIFFERENCES WITH DDB PHASE 1 In DDB Phase 1, when any DDB variable is updated, a checksum would be calculated for the particular entry and the table right away - e.g. if a link state changes, the link entry and the link table checksum would be immediately calculated. For performance reasons and for maintainability, DDB Phase 2 implements a background DDB checksum task that re-calculates checksums for the entry, and the table after the fact. In DDB Phase 2 (like in DDB Phase 1), the quiet time requirement must be met for an audit calculation to come up with a result of either consistent or inconsistent. An additional check will be done on per card basis (which will be described in more detail later.) If DDB Update in progress is TRUE, either DDB checksums have not been calculated completely on that card by the new DDB checksum task, or the (existing) SS7 TSRC task has not completed evaluating all routes in the route table. The first check (quiet time requirement) is mandatory and is required to be met. Once this is satisfied, the second check is used to determine whether a cards DDB checksums can/should be used for overall DDB checksum calculations and determination of DDB state. The DDB audit task that monitors and determines DDB state has the following two major functionality: 1. Dirty Bit audit (checksum calculation) (new in DDB Phase 2). 2. Wild Write Audit (WWA) (Updated the Phase-1 implementation) 2.2 DDB WILD WRITE AUDIT WWA has been modified in phase to run in everytimeslice with-in the audit task. The major functionality of WWA is as follows: 1. WWA will only work when the table under monitor is clean i.e. table dirty bit has not been set. 2. Check for checksum discrepancy for each table and table entry and rectify the checksum 3. In case WWA finds any table entry with the dirty bit as set then WWA will skip the entries for the dirty audit task. When a checksum is identified to be incorrect and is updated by a wild write audit, the user may want to know that a DDB inconsistency reported on a card was due to 9

a wild write (rather than any other cause, like missed multicast). Hence in DDB Phase 2, each MTP card will store the last 10 entries in chronologically reverse order which were updated by wild write audit on that card. These in-memory entries will get erased when the card reboots. The user can query these wild write entries by issuing the command: dbg-ddb=<card addr>:action=wwa (new parameter). Following information will be stored for each such entry. 1. On card time stamp (milliseconds since card initialized) when the checksum detected and updated. (At this time, the current date and time are not accessible readily on the MTP cards, in the future, this card specific time stamp could be replaced by a date and time). 2. Table ID/name (e.g. Link table, Linkset table, Route table etc.) 3. Entry ID (For tables which have no entries, only one table, the entry ID field will be blank) 4. Original checksum 5. Checksum after update by wild write audit task 2.3 NEW DDB TIMERS Timer DDB background audit timer Description No change in Phase 2: Time interval between consecutive DDB audits Value(s) 5 to 1440 mins Default: 10 mins 2, 5, 10 secs 4000 ms

DDB audit retry timer DDB audit replies timer DDB Audit time slice

No change in Phase 2: Time between retries when DDB idle period threshold is not met. There shall be only 3 retries. Changed from 2000 ms to 4000 ms in DDB Phase 2. Time within which all audit replies are received. (This should account for STH poll period) Time intervals between each time-slice of DDB audit process

5 ms

10

3.0 METHODS OF RESOLUTION OF DDB ISSUES

When a system report DDB inconsistencies t he first thing to check in the system are the IMT busses statistics in order to make sure they are clean. This point is really important since multicast updates resulting from network activity transit via the IMT busses and any outstanding issue on this part of the system may lead some cards to miss the updates and OAM to report DDB inconsistencies. Starting release 41.1 when a checksum is identified to be incorrect and is updated by a wild write audit, the user may want to know that a DDB inconsistency reported on a card was due to a wild write (rather than any other cause, like missed multicast). In the case one or a group of cards miss a DDB update related to a network state change then the counter collecting the total number of update misses is incremented. This counter can be retrieve either via a send-msg or a debug ddb command. The parameters to use in the dbg-ddb command is depending on the type of DDB update missed (Route/link/linkset ). For more details refer to paragraph 1.3. The more DDB updates one or a group of card would miss and the less chance the card would have to get a consistent DDB. The number of misses for a card could decrease if the same state change leading a DDB update to be multicast to other cards occurs and this time it is not missed by the card with its DDB inconsistent. This means that a card needs the same state changes that have been previously missed to recur in order for the total number of misses to abate till value 0 is reached and when this happens then the card would have its DDB consistent. Example 1 Consider the following scenario a card is missing for some reasons a DDB updates which are multicast through the IMT and then got out of sync. In this case based on this the new checksums computed for this card would be different than other cards which have received all updates and then this card would be reported as having a DDB inconsistent. To bring the card back the consistent state there can be many ways. Such as first finding out where the inconsistency is, such as if the inconsistency is in route table etc. Then there can be steps to rectify this problem. The below explains which option can be used in order to resolve DDB issues.

3.1 ADDING A SECONDARY ROUTE BY USING A DUMMY ROUTE TO TRIGGER A ROUTE STATE CHANGE
There is method to clean DDB checksum inconsistencies for route table checksums. This method is not to be communicated to the customer and has to be applied by experienced TAC engineer.

11

The method consists of adding a secondary route (or a route with the lowest cost if more than one route to the DPC already exist) using an APC which is in a prohibited status to a DPC. The DPC is given by the send-message result as displayed by the below command. The use of the prohibited APC to create the route does not disturb the current behavior of the system. When the route is created then the second step consists of deleting the route just after. When this done then run the send-msg again which should give another PC to which the method is to be applied and the totalMisses should decrement each time. Perform this while the send-message does not report that the route status is clean.

Example: With a send message the command would be run on card 1317 which report a ddb checksum issue for RTE table. send-msg:ds=1:da=h'e1:f=h'86:d0=2:loc=1317 With a dbg-ddb command by using card 1314 as a reference card in which no DDB checksum issue is reported

Dbg-ddb :loc=1317 :Tbl=RTE:action=AUD:audtype=UC:RLOC=1314

The commands will provide a total misses number and the first PC to which the route using a dummy linkset will be added

paristk1 09-01-06 11:53:27 FWT EAGLE [1317]Card->System (Tbl:2) Diff:Card[4217] LclCksm:h'65ad RmtCksm:h'63ad TblAuditDone:TotalMisses:h'1 FirstMiss:h'661 ; paristk1 09-01-06 11:53:27 FWT EAGLE [1317] [Route:h'661] Chksum h'b426 at h'76e04a (42 bytes) PC: 6-039-6 LstRt:0 CmbRt:6 Dyn:h'1 TFC:0 Xlst:0 MOBR:2 NAdj:1 3 3 3 3 3 NmTFR:0PrevStat:1 SRT:h'727db2 (26 bytes) AKT:h'896afc (12 bytes) ;

paristk1 09-01-06 11:53:27 FWT EAGLE [4217] [Route:h'65c] Chksum h'b226 at h'876d298 (42 bytes)

12

PC: 6-039-6 LstRt:0 CmbRt:6 Dyn:h'1 TFC:0 Xlst:0 MOBR:0 NAdj:1 3 3 3 3 3 NmTFR:0PrevStat:1 SRT:h'86ac7b8 (26 bytes) AKT:h'8744030 (12 bytes) ; 2/The next step should be adding a route to that PC 6-039-6 using a linkset which has its APC prohibited. The addition of the new secondary route will trigger a route state change for that point code and then a DDB update will be sent to the card 1317. 3/ the route should now be deleted and the above send-message/dbg-ddb run again to get the new PC to which the route should be added/deleted in order to continue decrementing the number of DDB update missed and clear the DDB inconsistency. As the total number of DDB update misses is 1 then this one would decrement to 0 as the card 1317 has now received the missed update. The card 1317 gets now its DDB consistent and DDB alarm is cleared. Note that this method is useful when the number of misses is quite reasonable otherwise it may take a long time to trigger all DDB updated missed and clear the DDB inconsistency.

3.2 CARDS REPORT DDB MISMATCH CHECKSUM FOR LINKSET TABLE The method is quite similar to the one explained in section 3.1. For example if some cards report DDB checksum mismatches then the method of resolution consists of running DBG-DDB command or via send-msg to identify the linkset from which the cards missed DDB updates. Actually the debug command still reports an adjacent point code number from which the linkset can be deducted. The below is an example of debug command that can be run to figure out the point code number in question and the total number of DDB update misses. The debug command use a reference card in loc 1314 and the card reporting the DDB mismatch for the linksets. The point code is 867-aa and the total number of DDB update missed by card 1313 is 1.

> Dbg-ddb:loc=1313:Tbl=ls:action=AUD:audtype=UC:RLOC=1314 tstpe0akb1 09-10-20 14:31:26 IST EAGLE5 41.0.0-62.33.0 Dbg-ddb:loc=1313:Tbl=ls:action=AUD:audtype=UC:RLOC=1314 Command entered at terminal #19. ; Command Accepted - Processing tstpe0akb1 09-10-20 14:31:26 IST EAGLE5 41.0.0-62.33.0 User Message sent to location 1313. ;

13

Command Executed tstpe0akb1 09-10-20 14:31:27 IST EAGLE5 41.0.0-62.33.0 Card[1313]->card[1314] (Tbl:1) TblAuditDone:TotalMisses:h'1 FirstMiss:h'1 ; tstpe0akb1 09-10-20 14:31:27 IST EAGLE5 41.0.0-62.33.0 [1313] [Linkset:1] Chksum h'aa68 at h'8d23787 (167 bytes) Assign:1 Avail:1 APC: 00867-aa ITUNVar:0 ; tstpe0akb1 09-10-20 14:31:27 IST EAGLE5 41.0.0-62.33.0 [1314] [Linkset:1] Chksum h'f287 at h'8d23787 (167 bytes) Assign:1 Avail:1 APC: 00867-aa ITUNVar:0 ;

In order to resolve this issue then the user need to find out which linkset uses APC 867-aa and next during a maintenance window deactivate all the links and activate them again to trigger a linkset state change. When this happens then DDB update missed would be generated and total number of misses decremented. If the total number of misses is different of 0 then this needs to be repeated until value 0 is reached and the DDB inconsistency and DDB alarm should be cleared. In some cases the full linkset deactivation/reactivation may not be sufficient to trigger the expected DDB update and the linkset may need to be cancelled/re-created as an alternate option. Modify a linket parameter (which will not impact the normal functioning of the linkset) and then revert it. (for example you may assign a gateway screening set in test mode to the linkset and then remove). Do aud-data:type=ddb:display=all to verify the DDB status 3.3 CARDS REPORT DDB MISMATCH CHECKSUM FOR LINK TABLE The resolution of DDB issues related to link tables is similar to the one explained in section 3.2 The cards reporting this issue are found via a rept-stat-ddb in full mode then a ddbddb command may be run to the card reporting DDB mismatch for link table as per the below example. This command will provide the link ID and the card from which a state change has been missed and the total number of misses. The link ID is mapped with link port as follows: A :0 , B:1, A1:2, B1:3, and so on Dbg-ddb :loc=1317 :Tbl=LNK:action=AUD:audtype=UC:
blmptss 08-10-07 15:40:08 FST EAGLE 41.0.0-62.33.0 [1317]Card->System (Tbl:0) Diff:Card[4101] LclCksm:h'e5e8 RmtCksm:h'6e8 TblAuditDone:TotalMisses:h'1 FirstMiss:h'1c ;

14

blmptss 08-10-07 15:40:08 FST EAGLE 41.0.0-62.33.0 [1317] [Link:x1c] Chksum h'3f52 at h'7b5c60 (14 bytes) PortId:h'656 (Card:2114 Link h'6) Slc:1 Stat:h'1 LsId:h'6 Class:0 Status:Avl

In this case a link deactivation/activation of 2114:A3 should trigger the missed DDB update and decremented the total number of misses from 1 to 0.

15

16