OMBUDSMAN FINANCIAL SERVICES

COMPASS PORTAL

INVESTIGATION Incident Report for Joint

Troubleshooting in slow upload files in
OFS System

Prepared By:

Solsis Sdn. Bhd

Ombudsman Financial Services 1

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Introduction
This document is about comparison of server specification used in client environment
(Production Server - http://compas.ofs.org.my/) and Solsis Team environment (Staging
Server - https://staging.ofs.dpay.my/).

User complaint that there is issue when uploading file when they using the system. IT Admin
OFS approach and suggest to put all the CDN,JS,CSS and etc to local as it seems there is
issue to access to the package.

Purpose
To compare server specification used for this project and give suggestion on how to improve
current problem.

Problem Statement
 User complained that usually it take up to 30 minutes of uploading file in the
production server.
 IT Admin suggest hosting the CDN into locally in the server production.

Ombudsman Financial Services 2

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Table of Contents

Introduction..........................................................................................................................................2
Purpose................................................................................................................................................2
Problem Statement.............................................................................................................................2
Action Taken........................................................................................................................................4
Server Comparison.............................................................................................................................5
Incident Report....................................................................................................................................6
Incident Joint Troubleshoot Process 16th February 2023..............................................................7
Results.................................................................................................................................................8
Effected Process Flow.......................................................................................................................8
Suspected Effect Process Flow........................................................................................................8
Corrective and Preventive Measure.................................................................................................9
Conclusion...........................................................................................................................................9

Ombudsman Financial Services 3

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Action Taken
 We have done server comparison for both staging and production environment to
check if it’s the environment issue. We already check and verify that the production
environment is much advance than staging environment.
 We already test to upload large files (> 100MB) into the staging server and we did
not encounter any issue.
 We already take upon their suggestion and put the javascript, css and etc into local.
 We have push and try into staging environment for the new request and passed
successfully.
 We perform Joint Troubleshoot in OFS to find the real cause of the issue on 10.30
AM till 12.30 PM at 17 February 2023.

Ombudsman Financial Services 4

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Server Comparison

Table below are some comparison between the staging server of compass and production
server. This information is gathered via several investigations through servers.

Comparison Table
Server Staging Production Production Conclusion
Details (192.168.1.120) Database
(192.168.1.121)
Hardware
Memory 2 GB 8 GB 16GB Production server is
advance than staging.
Processors 2 Core (1 socket, 8 Core(1 socket, 8 2 Core (1 socket, 2 Production server is
2 core) core) core) advance than staging.
Hardisk 25 GB 196 GB 196 GB Production server is
advance than staging.
OS Ubuntu 20.04.5 Ubuntu 20.04.5 Ubuntu 20.04.5 Production server is
LTS LTS LTS advance than staging.
Software
PHP 7.4.3 7.4.3 N\A Production server is
version advance than staging.
MYSQL 8.0.31- N/A 8.0.31- Production server is
version 0ubuntu0.20.04.2 0ubuntu0.20.04.2 advance than staging.
Linux x86_64 Linux x86_64
Laravel 8.83.25 8.83.25 N/A Production server is
version advance than staging.
Apache 2.4.41 ubuntu 2.4.41 ubuntu N\A Production server is
version advance than staging.
Network Cloud - Internet Local LAN Local LAN
Figure 1.0 Investigation of server performance

Ombudsman Financial Services 5

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Incident Report

Date Activity Remarks

10th Dec 2022 IT Admin OFS Ahmad reach

to us that user have issue on
uploading file.

12th Dec 2022 We have started investigation

on the issue and compare our
server detail and
configuration.

05th Jan 2023 We have taken upon their

suggestion and put all the
CDN, CSS, JS and etc to
local server.

08th Jan 2023 We pushed all the CDN,

CSS, JS and etc to our
staging server.

11th Jan 2023 We already tested and

confirmed no issue on
staging due to the change of
the CSS and JS file. We
pushed the code into
production server.

02th Feb 2023 The issue of slow upload file

is still persisting. We suggest
going to OFS and perform
Joint Troubleshooting with Mr
Jeff Guan and Mr Hisham
Aderis.

16th Feb 2023 We do Joint Troubleshooting

at OFS with their IT Admin.

Ombudsman Financial Services 6

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Ombudsman Financial Services 7

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Incident Joint Troubleshoot Process 16th February 2023

This is the full incident process list when doing Joint Troubleshoot Process 16th February
2023 when we doing the Joint Troubleshoot.

10.30 AM See user upload file through

system.

10.40 AM So far test upload file from

user 1 out of 7 is failed and 1
out of 7 take long time to
submit.

10.50 AM Check network on user

desktop to ping App Server
and DB

10.55 AM Check Physical Server and

Network setup flow by Jeff
Guan.

11.00 AM Create File from server and

export to db server = Passed

11.15 AM Transfer file from sftp using

FileZilla to App Server =
Passed

11.30 AM Check configuration in php.ini

for upload size configuration.
No configuration change
since all setting is on and set
okay.

11.45 AM Open debug mode for laravel

for testing.

12.00 PM Enable slow_query in

database.

Ombudsman Financial Services 8

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA
12.15 PM Test upload file size 114MB
through system in user profile
in function "Document Detail”.
Upload success in 15 second
in Production.

12.30 PM Test upload file size 114MB

through system in user profile
in Function "Pending
Document from FI" Upload
take longer times to process.

Results
1. Upload file issue is in certain process flow
2. Will find a workaround for current process to have smooth process meanwhile finding
resolution on this Issue.
3. Process flows that have issued with Upload File:
a. Pending Document From FI
b. Process that force to upload file(will find all the process that impact on this)

Effected Process Flow

1. Prepare first letter for FSP
2. Query Letter
3. Pending document from FI
4. Prepare acknowledgement letter
5. Waiting for required document from FSP
6. Waiting for required document (Query Letter)

Suspected Effect Process Flow

1. Amend recommendation
2. Email recommendation
3. Draft closure letter
4. Re-draft closure letter
5. Prepare settlement & agreement
6. Prepare settlement
7. Prepare Query Letter

Ombudsman Financial Services 9

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Ombudsman Financial Services 10

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Corrective and Preventive Measure

We suspected the issue on file creation when it enter the certain process. Therefore,
we will plan to make changes for one process flow and will do the testing at a live
environment. If the issue as we suspect is correct we will go with flow to change the process
for all efffected process and suspected process.

This is the process for our plan for this fix.

Phase Phase 1 Phase 2 Phase 3 Phase 4 Phase 5

Activties Fix one Deploy the If the issue We will deploy We will continue
process changes on is about all the bugfix the process for
and deploy production the file and be on troubleshooting
and test at for testing. creation at standby for 3 and will planned
staging (For this the certain days working out our plan for
process we process hours to this.
will come to then we monitoring the
OFS and do will deploy issues.
it after office the fix for
hour. This all the
process effected
need user to process
stop use for and
compas for proceed for
certain phase 4. If
period of the issue is
time) still persist
then
proceed to
phase 5

Estimated 3 Days 3 hours 14 Days 4 Days include TBD

Time Bugfixing Onsite of monitoring
issues

Ombudsman Financial Services 11

OFS INVESTIGATION TIME CONSUMING OF UPLOADING FILE

OFS – CEA

Conclusion
After series of troubleshooting and comparison making we have put our best effort to
minimize the factor of this issue. Therefore, we have found the issue is in the system
process of our application. We will find an alternatives or workaround for this process to this
issue before we will push the final resolution and fixed it. We are deeply sorry for this issue
and cause that this effect for the user and will give our best to optimize our application.

Ombudsman Financial Services 12