STC - 2013

Test Report Dashboard – Art of Showcasing data

graphically, dynamically

Prepared by: Indium Software India Ltd.

Name : Poornima Gopalan & Vishnupriya B
Email : poornima.g@indiumsoft.com
vishnupriya.b@indiumsoft.com
Contact # : 91-98409 81449, 91-90925 25934
 Test Report Dashboard – Art of Showcasing data dynamically, graphically 1
-------------------------------------------------------------------------------------------------------------------------------------------

TABLE OF CONTENT

1 ABSTRACT ........................................................................................................................................................ 2
2 WHY WE NEED................................................................................................................................................. 2
3 THE IMPORTANCE ......................................................................................................................................... 2
4 REPORT TYPES ................................................................................................................................................ 3
4.1 DAILY / WEEKLY / MONTHLY STATUS REPORT ............................................................................................ 4
4.1.1 Contents ............................................................................................................................................... 4
4.1.2 Report Designing ................................................................................................................................. 4
4.1.3 Benefits ................................................................................................................................................ 6
4.1.4 Recipients ............................................................................................................................................ 6
4.2 PRODUCTIVITY / PROJECT MANAGEMENT METRICS REPORT ........................................................................ 6
4.2.1 Contents ............................................................................................................................................... 6
4.2.2 Report Design ...................................................................................................................................... 7
4.2.3 Benefits ................................................................................................................................................ 8
4.2.4 Recipients ............................................................................................................................................ 8
4.3 OVERALL TEST SUMMARY ............................................................................................................................ 9
4.3.1 Contents ............................................................................................................................................... 9
4.3.2 Report Design ...................................................................................................................................... 9
4.3.3 Benefits .............................................................................................................................................. 10
4.3.4 Recipients .......................................................................................................................................... 10
5 CONCLUSION ................................................................................................................................................. 11
6 REFERENCES ................................................................................................................................................. 11
7 AUTHOR BIOGRAPHY ................................................................................................................................. 11

Indium Software | www.indiumsoft.com

 Test Report Dashboard – Art of Showcasing data dynamically, graphically 2
-------------------------------------------------------------------------------------------------------------------------------------------

1 Abstract
One of the important facets of software development life cycle is testing. Software testing is an area that is being
considered and given utmost importance in the world of fast changing technology. There are various stages at which
testing are done to ensure quality of delivery.

Reports are the basis for many processes used to manage and communicate results from business operations. In
business today, the reporting process is often incredibly complex. It can include reports derived directly from
information systems, custom reports developed from data warehouses, and reports based on spreadsheets that
compile various system and manual inputs, among many others.

Most key reports not only assist corporations in understanding pertinent financial information, but also are critical
components of the control framework of the organization.

2 Why we need
Visibility, Accountability & Transparency (VAT) are essential to any project/product team at different levels such as
Project/Product level, Team level, Task level and also at the Team members level. An effective status report is the
only way that can make sure that these attributes are met

We do send reports on a day to day basis to communicate the accomplishments / tasks performed for the day,
however as a product owner or product team member do we get a consolidated list of these at will? The Answer is a
big “NO”. The team has to either go through a bunch of status reports or wait for the test team to consolidate and
send a weekly status report / test summary report.

Our solution is to inculcate all the objectives of a daily / weekly / summary report into a single report which not just
reduces time to prepare but also helps the project team to have access to all the data that they seek for at their will.

3 The Importance
Is it sufficient to give a single line certification that, the product is ready/not ready for implementation?

A detailed test reporting should bring in transparency to test coverage / effectiveness / defect metrics / test
confidence which the customer would appreciate most. This could help the customer to look for alternate solutions
sometimes
An effective test report should highlight
 Test Scope for release / sprint / product
 Test Coverage
 Test Effectiveness
 Test Metrics
 Strengths, Limitations and Recommendations
 Impediments
 Key Issues with solutions
 Other aspects such as performance, platform dependence etc.
 Test Confidence

Indium Software | www.indiumsoft.com

 Test Report Dashboard – Art of Showcasing data dynamically, graphically 3
-------------------------------------------------------------------------------------------------------------------------------------------

4 Report Types
Testing teams send out various reports at different phases of the STLC,

It is enough to communicate with the rest of the project teams, when a test plan is created
Planning
or when a major change is made to it based on the overall product / project plan devised
Reports in these stages will highlight information on the High level test scenarios (HLTS)
/ Low Level Test cases (LLTC), Traceability Matrix (RTM), Test Data generation and
Designing other activities to all the project team stakeholders.

These will not only let them know about the progress of the task but also signal the teams
that need to review and provide signoff on the artifacts, that they are up next
Execution is the phase of a project when the testing team is the primary focus – positively
and negatively.

The test execution stage consists of:

Setting up the test environment as detailed in the test case specification
Selecting and running relevant tests from the test case spec
Recording test results
Execution Recording and managing defects
Final Reporting of the test execution phase

A typical day during a test cycle is not done, unless the daily status report is sent out. In
some teams, they could agree on a weekly report, but having it sent daily is the norm. It is
also not uncommon to have a status meeting everyday (or week) to present the QA team‟s
status to the concerned parties. Hence, the mode of a status report can be:
 Email/document
 Meeting/presentation

The following are the different types of test reports:

Daily / Weekly Status Report

Productivity / Project Management Metrics Report
Monthly Summary / Release (or) Sprint summary Report
Overall Release / Test Summary Report

Indium Software | www.indiumsoft.com

 Test Report Dashboard – Art of Showcasing data dynamically, graphically 4
-------------------------------------------------------------------------------------------------------------------------------------------

4.1 Daily / Weekly / MonthlyStatus Report

Status report templates are used by teams needing frequent check-ins against progress. Product development and
project management teams often use daily status reports (sometimes in the form of a scrum) to keep everyone
coordinated and on track toward team goals.
During the execution phase, test case results and any defects will need to be recorded in order to track and manage
the test progress until all planned tests are completed or the final acceptance milestone has been reached. This
information is often stored via a test management tool such as Quality Center or an Excel spreadsheet such as the
Test Execution Dashboard which contains the following

4.1.1 Contents
On a day to day basis, we provide information on the four parameters listed below
 What did we do today?
 What is for tomorrow?
 Impediments we faced with resolutions (if any)
 Dependencies
In addition, what we propose is to include contents of what exists generally in the test summary reports such as
 Test Scope – highlighting the features (or) areas in scope for the release
 Test Approach – Depicts the strategy planned for the release / test cycle
 Test Deliverables – Lists all the deliverables to be sent for review / submission
 Test Coverage – Indicating the progress made everyday (Designing / Execution stage)
 Test Effectiveness – Indicating status of each of the LLTC (Execution stage only)
 Defect Metrics - By Status / Priority / Severity / Module (or) Verticaletc. to name a few
 Key Issues with resolutions – This helps to conduct postmortem on what went wrong and how we resolved
the issue and serves as data point for future reference. E.g. capturing environment downtimes with reasons
 Test Resource Productivity – Capturing information on how much time was spent by each resource on
which task / activity to capture the root cause analysis on any schedule deviation or effort variance
The same report can include additional information such as Build Metrics capturing the number of release number or
version, number of features implemented, number of bugs fixed etc. to have more insight on the number of builds
worked on by the test team for a Product / Project Release

4.1.2 Report Designing

It becomes difficult when all the above mentioned has to be represented into a single component and still not be
looking as a troublesome process from an understanding perspective.
The below snapshot depicts how to represent the status report effectively and efficiently in MS Excel 2010 using
some of the key features such as Pivot tables & Slicers which help us to generate data dynamically.

Header: Includes the Client name & the Reporting Period, the current date is listed below, on the right side of the
header is a synopsis of the test approach

Middle Section: Includes the day wise activities along with the areas in scope & resources used. The right pane of
the report includes graphical representations of the overall defects in the product / project along with the slicers
which allow us to filter and view the graphs dynamically based on the selection. For instance the user can choose to
view the current list of defects that are critical and active. This can also support applying multiple filters

This section also includes areas for deliverables in scope, action items/ Impediments and also helps to capture key
issues with resolution notes

Indium Software | www.indiumsoft.com

 Test Report Dashboard – Art of Showcasing data dynamically, graphically 5
-------------------------------------------------------------------------------------------------------------------------------------------

Fig.1 Daily Status report Dash board

Bottom Section: Includes Test design metrics on how many High level test cases are designed by module or vertical
and how many low level test cases were drafted for the HLTS identified. This can include the coverage /
effectiveness graphs in the execution phase and can have slicers for the users to filter and view the report

Fig.2 Test Case Completeness by Module

Fig.3 Test Coverage & Test Effectiveness

Indium Software | www.indiumsoft.com
 Test Report Dashboard – Art of Showcasing data dynamically, graphically 6
-------------------------------------------------------------------------------------------------------------------------------------------

Test Coverage:A measure of the proportion of a program exercised by a test suite, usually expressed as a
percentage. This will typically involve collecting information about which parts of a program are actually executed
when running the test suite in order to identify which branches of conditional statements which have been taken.

Test coverage matrix determines the test completeness. It measures the percentage of test execution details with
respect to the no. of high level cases designed

Test Effectiveness: This is a metric to captures the effectiveness of the test cases executed based on the number of
passes/ failures of the documented test scenarios. This determines the test confidence on the application.

Execution & Coverage reports answer the following questions:

How much work is complete, how much work is left, and is progress being made as expected?
Which module / vertical have high numbers of failed points or test case execution records?
Which module / vertical plans have high numbers of defects?
Which systems have high numbers of failed points or test execution records?
Which test case categories have high numbers of failed points?

Monthly Status reports shall include consolidated weekly sheets, wherein we pull in data from different weeks and
create a data sheet from which project team can pull information on any specific week or release.

4.1.3 Benefits
The objective of capturing this data on a day to day basis is to
 Save timeby not having to spend repeated efforts of preparing multiple reports through consolidation of
your daily tasks. Instead updating this sheet on a daily basis converts the same into a weekly status
report with few additional sections and minimal effort
 Enhance Tracking mechanism by bring in Transparency & Visibility on the progress made from the
designing stage to the execution stage and thereafter as well
 Improve Communication by communicating risks / issues on a regular basis
 Improves efficiency by analyzing the productivity / root cause analysis on the delays

4.1.4 Recipients
Generally, the recipient of this email/report is generally the manager, also the team members can be CC‟ed in some
cases – this depends on the communication protocol the team follows but we recommend this to be made available in
a repository (Cloud / Server) for any of the project stakeholder to have access to the document at his/her own will.

4.2 Productivity / Project Management Metrics Report

The core objective of this report is to get insight on the productivity of the team members on various tasks /activities
performed in different stages of the STLC which can be used to transform how we can manage resources and derive
sustainable growth in future which can help the organization.

4.2.1 Contents
In today‟s fast world, getting timesheet information filled is considered as strenuous ask , developing an internal
application or procuring one from market, customizing , maintaining is even more challenging. To avoid all such we
developed a simple 4 step process to record the task / activities executed by the team members with the timelines.

The productivity report will include the following

Day wise task/ activities with the time spent

Schedule Deviation with RCA
Effort Variance with RCA
Milestones Achieved
Upcoming Milestones
Top Risks with mitigation plan

Indium Software | www.indiumsoft.com

 Test Report Dashboard – Art of Showcasing data dynamically, graphically 7
-------------------------------------------------------------------------------------------------------------------------------------------

4.2.2 Report Design

Summary sheet will include project specific information such as Lead, customer contact,Domain, start & end date
along with resources allocated by module / verticals. The productivity of the resources will be captured as a
consolidated list against modules if there are multiple resources working on the same module.

Additional information such as the number of billable / shadow resources and new resource request shall also be
comprised as part of this section

Fig.4 Project Management Metrics Dashboard

The second half of the report includes the Delivery related metric such as capturing the planned deliveries against the
actual deliveries made and includes information on the root cause of the deviances (if any)

Fig.5 Effort Variance

Some of the common RCA‟s are as listed below

RCA
Change Request Coding Issue Coding Standard Issue
Analysis Issue Availability Change in Priority
Configuration CR Data Issue
Design Issue Duplicate Issue Environment issues
Estimation Framework Issue Incompetency
Not an Issue Planning Skill
Unclear Requirement

Indium Software | www.indiumsoft.com

 Test Report Dashboard – Art of Showcasing data dynamically, graphically 8
-------------------------------------------------------------------------------------------------------------------------------------------

The second sheet in this report includes the detailed productivity hours per resource as depicted below. This serves
as an input to the productive hour‟s column in the summary sheet

Fig.6 Productivity Hours Table

The last section of the summary report will include additional metrics specific to the team members and their
contributions towards the project on weekly basis or activity wise

Fig.7 Weekly basis Metrics

4.2.3 Benefits
Helps organization to identify the effective productivity of resources
Helps in analyzing areas where the project team had spent long hours and helps improve resource
optimization and planning for future
RCA helps the customer and the project team in determining the milestones for future releases
Improves test estimation for future projects with similar domain/ technology
Helps customer to realize ROI for any kind of delivery engagement esp. T& M projects as this brings in
transparency on the output of the individuals in the team

4.2.4 Recipients
The recipient of this email/report is to all project stake holders Internal & Customer. This report can also be
maintained in the repository (Cloud/ Server).

Indium Software | www.indiumsoft.com

 Test Report Dashboard – Art of Showcasing data dynamically, graphically 9
-------------------------------------------------------------------------------------------------------------------------------------------

4.3 OverallTest Summary

Test summary report will be different for different kind of testing. If the project is risk based testing then the report
should contain the test coverage and unresolved defects against the product risks, and if it is requirements based
testing then the test summary report should measure coverage against the requirements.
In addition to test coverage and unresolved defects test summary reports should also contain test strategy, test
objectives and overall result of test effort.

4.3.1 Contents
Test Summary Report Includes:
Objective
Test Scope
Variances
Comprehensive assessment
Evaluation
Executive Test Summary
Our Challenges & Findings
Project Metrics
Project Deliverables
Summary of activities
Summary of results
Next Steps
Approvals

4.3.2 Report Design

Test Summary report can be best showcased in MS presentation. It shall contain following sections along with
pictorial representations.

Objective: Includes the testing focused areas, system requirements and features in scope for the testing.

Fig.8 Test Summary - Objective

Test Process: Includes the process that the QA team followed to streamline the application under test and to enhance
the test effectiveness. Also includes analyzing of test Results, to determine the effectiveness of testing that
QAperforms

Challenges and Key findings: Includes the challenges we faced during this Sprint/Release. The Major issues of this
Sprint/Release will come under Key findings.
Indium Software | www.indiumsoft.com
 Test Report Dashboard – Art of Showcasing data dynamically, graphically 10
-------------------------------------------------------------------------------------------------------------------------------------------

Value Adds:Includes the recommendations / suggestions or any other „extra‟ features beyond the standard
expectations that might have given good ROI. This can also include following:

Total number of defects raised

No. of recommendations given

Defect Metrics:Includes the overall project defect metrics in a pictorial representation based on following
categories, in the same format as DSR

i. Status
ii. Priority
iii. Severity
iv. Bug Type
v. Module / Component wise
vi. Also, the statistics of defects based on two categories can be given as in following graphical representation

Fig.9 Test Summary – Resolution vs. Priority

4.3.3 Benefits
There are many benefits that can be gained by using Test Reports. They are:
Clear visibility of work in progress
Reduction of time spent
Greater consistency and repeatability
Objective assessment
Ease of access to information about tests or testing
Helps development team to achieve better quality in future releases
Helps customers with key decision making with respect to release of products

4.3.4 Recipients
The recipient of this email/report is to all project stake holders Internal & Customer. This report can also be
maintained in the repository (Cloud/ Server).

Indium Software | www.indiumsoft.com

 Test Report Dashboard – Art of Showcasing data dynamically, graphically 11
-------------------------------------------------------------------------------------------------------------------------------------------

5 Conclusion

Test reporting has no end to the value additions that one could provide with good report dashboard. But it is
prominent to trail test report from start of any project until its closure
A dynamical and accurate test report dashboard helps its recipients obtain better visibility for key decision
with respect to release of products
Dynamic test reports can be showcased well in MS Excel along with graphical data
All future risks, mitigation & contingency plan can be arrived at by tracking pertinent test reports. Future
milestones for any project specific proceedings also stands as „easy to arrive‟ with virtuous report
dashboard

6 References
http://istqbexamcertification.com/what-is-test-coverage-in-software-testing-its-advantages-and-
disadvantages/

http://www.softwaretestingmentor.com/test-deliverables/test-execution-reports/

7 Author Biography
Poornima Gopalan has an overall IT experience of 9 years on delivering IT solutions on several technologies over
different domains through Software testing. She is a Quality Analyst in Indium Software and was associated
previously with Cognizant Technology Solutions. Her current work includes coordinating and managing multiple
test teams to deliver quality at its best. She has passion in training testers to go beyond test functionality and
coverage. Her interests also include test reporting and metrics. She is also passionate about Professional
Management Professional and a member of pmhub.net

Vishnupriya comes with 2.5 years of experience and ISTQB-BCS certified professional. She started her career as
Assistant Engineer QA in SmarTek Consultancy Services and worked in various projects related to mobile
technology. Currently she is working in Indium Software as Test Engineer for mobile application testing. She did her
Bachelor degree in Electrical and Electronics from Anna University, Tiruchirappalli. Her hobbies are listening music
and crafts work.

Indium Software | www.indiumsoft.com