First Vietnamese QA Firm

LQA – Toward the Perfection 1

 Software Test report
Case study – Test metrics

PRESENTED BY: THAM NGUYEN

HA NOI, FEB 2017

LQA – Toward the Perfection 2

Test case distribution
Test cases backlog
• Test Cases are organized though 2
categories covering the entire 46%
54% (160
integration of Platform, from Portal (187 TC)
TC)
to Backend…

# TCs Backend # TCs Portal

• As of today, about 50% of Test Cases in the Backlog are stable and considered in each Testing
Campaign. The remaining half concerns Test Case is either in Draft, under review or Obsolete.
Portal Backend

9% 9%

42% Validated Validated

Draft 56% Draft
54% Under Review Under Review
Obsolete 26% Obsolete

4%

LQA – Toward the Perfection 3

Test case category
• Validated Test Backlog remained stable from Jan to March. Increasing trend
starting from March driven by the integration of 2 new major building blocs (Apr
Release + May Release)
Test bank Growth Trends
Test bank Growth Trends
Total
168
152

68 107
99 99 99

68 68 68
55
33

22 22 22 23 22

Jan Feb Mar Apr May

Jan Feb Mar Apr May
Critical High medium Low

• Apr release
• May Release

LQA – Toward the Perfection 4

Test Coverage – Component wise

• Test Coverage reaches around 60%. Testing activities related to Payment Gateway is ongoing,
while updating some Test Scenarios with new behavior of previous functionalities …
Overall Test Coverage
Coverage by component
42% # Uncovered # Covered

58% 31
51
59
3
# Covered # Uncovered 20 27

Automation Coverage Trends

• As of May, 50% of the overall

Automated Total Test Cases
349 349
Backlog

331

Validated Test Cases are automated 252

268

and part of the on Regression Test

Campaign for new major release or 152 159 168
90 90 92 117
85 78 84
maintenance changes. Jan Feb Mar Apr May

LQA – Toward the Perfection 5

Test Efficiency – Defect Trend
• High efficiency in finding defects in UAT. Between Jan and May, defects represent between ~20% and
~40% of total Test Cases, and linked to impact of new functionalities, changes in requirements, and
hot fixes in target UAT environment …

Detected Defect Trend Detected Defect Trend

Failed Executed Failure ratio (%)

135
38,7%
123
31,9%
106 28,6%
98
22% 23,5%

43 41
35
27 23
10

Jan Feb March Apr May 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5

LQA – Toward the Perfection 6

Test Efficiency – Defect Severity
• Defect Severity distribution is tracked enabling teams to focus efforts in the right area of concerns
targeting Critical and Serious defects against cosmetics defects and enhancement requests.
Defects Severity
• 20%, of detected defect have been considered as Serious or
16% 9% 12%
Critical Critical either because the defect was affecting the
Serious
Medium complete integration chain or relating to unusable
Minor
63% functionalities.

Defects status
• Only 25% of the defects have been rejected mainly related
Accepted
26% 8%
10%
Discussion to Test Cases describing Layout or related to changes in
In progress
done requirements.
2% To be Tested

53%
Resolved
Certified • 10% are under discussion with Product Owner.
rejected

LQA – Toward the Perfection 7

Cost Distribution of Testing
Testing effort by feature
• Testing Effort is allocated equally 6%

between UI Postal and Back End 45%

ensuring a maximum test coverage at

49%
each level of the integration chain.

• Effort distribution is also tracked to help identifying intensive effort areas and target
improvement actions. Cost of Testing -- Phase-wise

Req review
15%
• 4% 12%
About 80% of the total effort focus in 4%
2% Writing test script

Writing manual TCs

operational activities. Test Execution

• Remaining effort relates to management tasks, 20%

30% Review

12% Test Report

reporting, and synchro with Product Owner Env setup

Meeting/Reporting/Mngt

LQA – Toward the Perfection 8

Effort Saving Through Automation
• Benefits of test automation practice start to be seen each month with a gain on the testing cost, as
well as a fast delivery of Test campaign execution and results.
Effort - Manual test Effort - Post Automation Effort Savings (men/days) Effort - Manual test Effort - Post Automation Effort Savings (men/days)

70 90
60 80
70
50
60
40 36 46 44
50
29
30 40
20 30 25
20 16 18 21
13
20
10 10
0 0
Jan Feb Mar Apr May Jan Feb Mar Apr May

 Effort Savings 2015:  Effort Savings 2015:

• 104 days  5 men/month between Jan and • 141 days  7 men/month between Jan and May
May • 2 regression test per month.
• 1 regression test per month.
 Limited Test Execution Effort in Feb and March:
• 2 new major Building Blocs (Apr release and May release):
• Requirement Review
• Test Case Writing
• Automation Scripting
LQA – Toward the Perfection 9
Effort Saving Through Automation
• Automation Cost is starting to be covered by the cost savings generated by running the automation
scripts in each testing campaign (based on 2 regression test per month, 2 scripters)

ROI of Test Automation

46 44
40 40 40 40 40

25
21
18
Cost of Automation (men/day)
Cost Savings through Automation (men/day)
6 4 Net Cost Savings (men/day)
Jan Feb March April May
Cumulative ROI
-15
-19
-22

-37
-46
-50
-56

• Automation return on investment will continuously increase shortly since requirements and
target environment are more stable. Positive ROI expected by end of this year.
LQA – Toward the Perfection 10
QnA/Discussion

LQA – Toward the Perfection 11

LQA – Toward the Perfection 12
REFERENCE
 http://www.istqb.org/
 ..

LQA – Toward the Perfection 13

Contact Us!

Website: http://lqa.com.vn
Email: hello@lqa.com.vn
Phone: +84 90 414 0881
LQA – Toward the Perfection 14