Luxoft testing practices for GCT

IT projects
Goals

§ Time to market
§ Efficiency - no redundancy on test case
execution
Challenge: How do we measure?

§ How do we normally test?

§ We use test cases
§ We use unit tests
§ What do we measure – code covered by unit
tests
§ What we can’t measure – what is tested by
test cases
§ Could we measure how much code is actually
tested?
§ Could we measure how much ‘dead code’ we have?
Measurement

§ What is measurement?
§ Is it number of test cases?
§ Is it how many days we spent for ad-hoc testing?
§ Is it number of days we spent to create/review test
scenarios?
§
§ Each test case covers some limited number
of source code lines
§ We could measure how much source code lines
(SCL) were used by particular test case
§ Using numbers of SCL by each test case we could
measure how much untested code we have,
§ We could identify unused code by not-covered by
test cases
Example

§ Project with 210 source code lines (Java/JavaScript/Ajax/etc)

§ Test plan has 5 test cases


 

 Regular scenario (guessing  Scenario with Source Code

style) Coverage info
§ 

§ Potentially we have § We have complete set of

coverage for 210 lines
(no instrument to
check)
§ Hopefully all source code
lines covered by test
cases
§ Potentially we have 
source code lines
tested by each test
case
§ We have complete set of
source code lines that
are not testes

somewhere 10 lines of
not tested code?
How to identify test cases
needed to check all changes in
the code

Not tested code

Test Cases to Execute

Test cases
Standard test case
 Proposed test case


§ How many steps do you have in § A few steps

your test cases? 30-200? § No dependencies – data required
§ Each step depends on previous to test functionality is created
steps in database right before test
§ How many times you test same case initialization
code using same conditions? § Test case tests only a piece of the
§ If 25’th step fails – how you’ll test very specific requirement
it again? § Can be used for both manual and
automation
§ Easy to support/understand
§ We call it Atomic test case
At om ic t e st ca se is:
§Scenario for very specific feat ure
§Scenario t hat is not relat ed t o any ot her t est case
§Original t est case is num ber of m any at om ic t est cases
§Each t est case has it s own coverage by source code lines
Atomic Functional Test Cases

§ Atomic test cases that aimed to verify

particular functionality
§ Benefits
§ Coverage: measurable
§ Maintainability: extremely high maintainability.
Case covers very small (atomic) piece of
requirement
§ Execution: easy to execute. Every case could be
executed separately even in different days by
different test engineers
§ Dependency: test cases are independent. Every
case prepares preconditions (application, data) for
it’s run
§ Automation: extremely easy to automate
Another example:
Conventional Test Case
Document for atomic test case
Traceability matrix
(functionality+testcase+source
code)
§Tra ce a b ilit y lin k s t og e t h e r
§Issue for new funct ionalit y wit h specificat ion docum ent
§Specificat ion docum ent wit h funct ional t est case
§Funct ional t est case wit h aut om at ed t est script
§Coverage for source code lines by test case
§“ Bla ck b ox” t e st cove r a g e b y m a n u a l/a u t om a t e d t e st s
Benefits

§ We won’t test few times same code

§ We reduce number of cases to test
§ We won’t release untested code
§ We identify what wasn’t tested
Automation platform for Rich UI
Implementation details

§
§
§ Tools: Selenium/WebDriver + TestNG
§ Luxoft’s developed Test Case Coverage
Tool
Test case pre-requisites
 To start with a test case we have to create data to test

Regular way
 Proposed way:


§ Create all data manually § If we have one atomic test

using standard flows case for data creation – we
using application UI don’t need to test it again
§ Depends on test case § We can pre-populate data in
complexity, may take database using some tool
about 30 minutes § XML based database
fixtures
§ Data that should
be already
present in
database to
pass test case
§ XML based GUI fixtures
§ Data that should
be entered into
form to pass
test case
Atomic test case execution

§ As plug-in to Maven


§ Atomic test cases for per-commit or nightly builds

§ Pre-requisites creation takes milliseconds
§ Test cases can be run against any browser without
changes


§ Performance
§ Individual atomic test case takes about 90 sec.
§ About 480 test cases per night
How does it work?
Coverage tool

Manual TC
execution Browser App DB

Auto-test-cases Coverage
tool

§Aut om at ed t est cases (using WebUIint eract ion API)

or QA m anually int eract wit h applicat ion wit hin
Coverage t ool t hrough browser
§Coverage t ool t racks t he source code lines affect ed
by t hose act ivit ies
 Cove r a g e t ool – colle ct in g
cove r a g e

Step Action

1.1:1.2 User/WebDriver activities

2.1:2.2 Communication with server

3.1 Cobertura tracks the code execution

4. 1:4.2 Dev Plugin controls the code execution on

Code Server

5.1 Emma tracks the code execution

6.1 The stop-tracking command request

7.1:7.2 Get client side coverage

8.1:8.2 Get Emma runtime coverage

9.1:9.2 Get Cobertura runtime coverage

10.1:10.2 Persist combined coverage for an activity to

DB
Benefits

§
§
§ We can measure the code covered by test cases
being ran either manually or by automated tool
§
§ We can determine the code not covered by test
cases
§
§ We can reduce the testing time by executing the
only test cases against affected source code
3rd party projects testing

§ Testing of 3rd party project?

§ No documentation?
§ No understanding how to test?
§ Quality is good enough?
§ Start with coverage, start with test cases
§ Having coverage for any test case, you can extend you
test coverage using tool for measuring
§ You can add more and more test cases for another areas
§ Future steps
§ You can MEASURE how many test cases affected (comparing to
coverage for previous version)
§ You can UNDERSTAND how much code lines are not covered by
your test cases
§ You can EXPLAIN how much dead code is in the project
§ You can ASK developers to explain how to test their code
§

And again all together

§
§
§ Atomic test cases
§ Test case code coverage
§ Minimizing of number of test cases required to
test all changes
§ Automated or Manual
§ Measurable!
Q&A