STRESS TEST OBSERVATIONS

CPU Utilization: CPU usage had reached to a maximum of 44.5% and rest of the time the usage looks
normal.

Memory Utilization: Memory usage was normal during stress test, and it never crossed 68% of total
memory.
No of FLEXCUBE Sessions

No of FC sessions for every 15mins interval.

No of FC
Time Sessions
5:00PM 20
5:15PM 97
5:30PM 249
5:45PM 701
6.00PM 746
6:15PM 682
6:30PM 604
6:45PM 563
7:00PM 501
7:15PM 305
7:30PM 209

No of ACTIVE Sessions

No of Active sessions for every 15mins interval.

No of
Time ACTIVE Time Node1 Node2
Sessions
5:00PM 13 5:00PM 6 7
5:15PM 26 5:15PM 16 10
5:30PM 38 5:30PM 20 18
5:45PM 129 5:45PM 61 68
6.00PM 97 6.00PM 49 48
6:15PM 89 6:15PM 46 43
6:30PM 102 6:30PM 48 54
6:45PM 94 6:45PM 42 52
7:00PM 96 7:00PM 50 46
7:15PM 92 7:15PM 38 54
7:30PM 100 7:30PM 48 52

MAFACE session count: Sessions are always high on Node 2 for this user.

MAFACE Node1 Node2

5:15 PM 7 6
5:30 PM 5 6
5:45PM 18 25
6:00 PM 15 20
6:15 PM 22 20
6:30 PM 26 27
6:45PM 31 36
 7:00PM 28 30
7:15 PM 24 40
7:30 PM 36 43

XAPP session count: XAPP sessions looks normal, and we didn’t see any XAPP sessions with more
count.

Blocking sessions: The below blocking session has been observed during the stress test and it got
released in few seconds.

Standby sync status

FCXPPT: This standby was in sync with Exadata FCXHO.

FCSSD: This standby was in sync with Exadata FCXHO.

FCPPTOFF: This standby was in sync with Exadata FCXHO.

Load Average on both the nodes: Load average during the stress test for every 15mins interval.

Time Load avg Node1 Load avg Node2

5:00PM 0.86, 0.70, 0.77 1.26, 0.92, 0.84
5:15PM 6.66, 7.23, 5.02 7.50, 7.18, 5.10
5:30PM 8.45, 8.03, 6.77 12.38, 10.66, 7.96
5:45PM 55.05, 43.60, 27.48 56.27, 51.10, 33.06
6:00PM 28.93, 36.18, 33.88 31.63, 38.92, 37.78
6:15PM 36.62, 37.85, 36.63 40.90, 41.78, 40.28
6:30PM 35.06, 37.41, 37.31 48.19, 47.69, 44.84
6:45PM 40.20, 40.29, 39.92 49.80, 52.73, 50.26
7:00PM 34.17, 38.08, 39.45 49.82, 49.06, 49.35
7:15PM 32.52, 34.80, 37.19 50.99, 50.43, 49.82
7:30PM 34.17, 38.08, 39.45 45.02, 45.68, 47.17

Instance Availability: Both the instances were available during stress test without any issues.
Redo Generation during Stress Test:

Redo generation while stress test is 164 GB.

Alert log: Alert logs were monitored for both the instances and didn’t see any issues.

No Timeout error occurred in database alert log file.

FRA USAGE: It is being monitored continuously and it was under threshold values.

Tablespaces Usage: Tablespace usage has been monitored and everything was under threshold
values.

AWR review of stress test -

Node 1 - Total cpu count on server is 128. Usage of cpu has maximum gone to 46.5 cpu per second.
Enough cpu is available on server.

Node 2 - Total cpu count on server is 128. Usage of cpu has maximum gone to 56.9 cpu per second.
Enough cpu is available on server.

1) Foreground Wait Event -

Network Wait class event - TCP Socket (KGAS) is still being observed in few AWR reports:

Solution - From the database point of view, these waits can safely be ignored; the wait event does
not represent a database issue.
This wait event is significantly improved from last runs.

From an application or network point of view, delays in establishing a network connection may
produce unwanted delays for users. You should make sure that the application makes network calls
efficiently and that the network is working well such that these delays are minimized.

2) Background Wait Event:

Ges Generic Event wait is observed in all AWR reports and Oracle support has already given patch
for this. This can be ignored and later can be supressed via patch.

3) Enqueue Row Lock Contention not observed in database:

No enqueue observed in this stress test. SQL_ID 1yu5ugmpzy019 is resolved and issue is not
appearing now.
UPDATE FCTB_ONLINE_STATS SET MAXRESP = :1 ,MINRESP =:2 ,AVGRESP= :3 ,COUNT =:4 ,
LOGTIME=to_date(:5 ,:"SYS_B_0") WHERE FUNCTIONID = :6 AND ACTION = :7 AND SOURCE = :8

4) SQL Order by Gets:

The below sql_id’s has high buffer gets in most of the AWR reports. Tuning these sql_id’s will
give good benefit to execution time in db. Please find the recommendation for one of the
SQL_ID.

Recommendation (estimated benefit: 99.97%)

------------------------------------------

Ran tuning advisory for mobapp query and sql profile is available. Bank needs to check this if this
can be accepting as recommended SQL profile.

execute dbms_sqltune.accept_sql_profile(task_name => 'sql_tuning_task_6gayy0mgcwcm7',

task_owner => 'SYS', replace => TRUE);
5) SQL Ordered by Elapsed Time:

Please find the below sql_id where it is being observed in most of the AWR reports and tuning
this will help in the performance of the DB.

Bank Team needs to check if 1st sql_id from mobapp can be tuned from the above screen
shot.

6) This query has been running for 40 min and was killed after bank confirmation.

select a.trnx_brn,a.user_ref,a.status,a.fcm_comment,a.fcm_reason_code,a.rej_by_c
orresp,a.is_vip,a.trnx_amount,a.trnx_ccy,a.account,a.acc_name,a.benname,a.benacc
,a.benaddr,a.benbankoption,a.benbankswift,a.benbank1,a.fway,a.purpose,a.details1
,a.details2,a.details3,a.trnx_fee,a.trnx_cable,a.trnx_senderfee,a.ira_acc,a.nost
ro,a.initiated,a.initiator,a.review_doc,a.review_customer,a.preprocessed,a.pre_p
rocessor,a.processed,a.processor,a.error,a.ftid,a.id from VW_OUTWARDS_SCRP_PBL a
where rownum <= 200 and processor='RA.BORIN'

Bank team needs to check this view query.

7) Blocking session:

Below blocking session has been observed and it got released after few mins.
 Sql_id - 'ajj93bj9wws1s'
UPDATE TEMENOS_OTT_REQUEST_STS A SET A.UDT = SYSDATE, A.CHECK_STATUS = :B6 , A.S
TATUS = CASE WHEN :B6 IN (:B5 , :B4 ) THEN 'OK2' ELSE A.STATUS END, A.FCM_COMMEN
T = NVL(:B3 ,A.FCM_COMMENT), A.FCM_REASON_CODE = NVL(:B2 ,A.FCM_REASON_CODE)
WHE
RE A.ID = :B1 RETURNING A.STATUS INTO :O0

Sql_id - 'a3af41ytu6vhb'
UPDATE OUTWARDS A SET A.STATUS = 'R' WHERE A.ID = :B1

Bank team needs to check this query logic to resolve Enq:TX – row lock contention.

8) LOV Queries:

Below LOV query has been observed and killed as it was taking time. It didn’t appear after
that.

5hs1r3jmbgg0c - SELECT * FROM ( SELECT Lovalias.*,rownum Rno FROM (SELECT * FROM

( SELECT BRANCH_CODE, CUST_AC_NO, AC_DESC, CCY, CUST_NO, NULL PHY_ACC, NULL
PHY_NAME FROM STTM_CUST_ACCOUNT A WHERE RECORD_STAT = 'O' AND AUTH_STAT =
'A' AND ACCOUNT_TYPE <> 'Y' AND ACCOUNT_CLASS NOT IN (SELECT VALUE FROM
LMTM_TYPE_VALUES WHERE TYPE = 'OW_CLEARING' AND CODE = 'ACLASS') AND NOT
EXISTS (SELECT 1 FROM STTB_ACCOUNT WHERE AC_GL_NO = A.CUST_AC_NO AND
(AC_STAT_NO_CR = 'Y' OR AC_STAT_FROZEN = 'Y')) UNION ALL SELECT PHY_BRN
BRANCH_CODE, VIR_ACC_NO AC_GL_NO, VIR_ACC_NAME AC_GL_DESC, CCY AC_GL_CCY,
VIR_CUST_ID CUST_NO, PHY_ACC, (SELECT AC_DESC FROM STTM_CUST_ACCOUNT WHERE
CUST_AC_NO = A.PHY_ACC) PHY_NAME FROM STTM_VIRTUAL_ACCOUNTS A WHERE
RECORD_STAT = 'O' AND AUTH_STAT = 'A' AND STATUS = 'A' ) WHERE (CUST_AC_NO)
LIKE :1 ORDER BY 1 ) Lovalias ) WHERE Rno > 0 AND Rno <= 25

Oracle team will check if history of this query is available in current production and tune it.

9) Long running sessions:

Below query has been observed with 3 sessions, these sessions were killed post 30 min after
confirming from bank.

8p09sy77zcbvq - select a.batch_id,a.name,a.status,a.obj_type,a.maker,a.dt,a.checker,a.cdt

from VW_UPLOTTRESTNM_DTL a where rownum <= 200 and batch_id='1573922'

Bank team needs to check this view sql.

10) Long running query : -

Below query has been observed and these were reported to bank as it was running for more
than 10 min.
 Sql_id - 734q4c5vy86k8

select
a.trnx_brn,a.user_ref,a.restrict_status,a.FCM_STATUS,a.FCM_COMMENT,a.FCM_CHK_DT,a.t
rnx_amount,a.trnx_ccy,a.rej_by_corresp,a.is_vip,a.is_waive,a.account,a.acc_name,a.benna
me,a.benacc,a.benaddr,a.benbankoption,a.benbankswift,a.benbank1,a.fway,a.purpose,a.de
tails1,a.details2,a.details3,a.trnx_fee,a.trnx_cable,a.trnx_senderfee,a.interswift,a.inter1,a.ira
_acc,a.nostro,a.initiated,a.initiator,a.review_doc,a.review_customer,a.preprocessed,a.pre_p
rocessor,a.processed,a.processor,a.error,a.ftid,a.id from VW_OUTWARDS_HSWIFT_PBL a
where rownum <= 2000

Bank team needs to check this view sql.

11) Long Running Query –

Below query has been observed and it was reported to bank as it was executing for more
than time.

Sql_id – 85w4u860mhb16

select
a.trnx_brn,a.user_ref,a.status,a.fcm_comment,a.fcm_reason_code,a.rej_by_corresp,a.is_vip
,a.trnx_amount,a.trnx_ccy,a.account,a.acc_name,a.benname,a.benacc,a.benaddr,a.benbank
option,a.benbankswift,a.benbank1,a.fway,a.purpose,a.details1,a.details2,a.details3,a.trnx_f
ee,a.trnx_cable,a.trnx_senderfee,a.ira_acc,a.nostro,a.initiated,a.initiator,a.review_doc,a.revi
ew_customer,a.preprocessed,a.pre_processor,a.processed,a.processor,a.error,a.ftid,a.id
from VW_OUTWARDS_SCRP_PBL a where rownum <= 20
0 and processor='RA.BORIN'

This query came from XAPP owner and package body - CVREQ$ACTION_STATIC_V2

Bank team needs to check this view sql.

EOD Details:

Posting 15 M entries took 25mins with 100 sessions on both the nodes.

EODM sql text has been changed so the sql_id got changed and ran with a different plan. We
will check the best available plan and baseline the plan.

Here is the EOD Timing.