DESIGN AND EMPIRICAL COMPARISON

OF MULTIPLE TEST ORACLES FOR GUIs

Ishan Banerjee

Masters Thesis
08/01/2003

Advisor: Committee Members:

Atif M. Memon Adam Porter
Marvin V. Zelkowitz

Department of Computer Science

University of Maryland
1

WHAT IS GUI TESTING ?

This is the text.

SelectText SelectText
Format Font 18 OK
(“This”) (“text”)

Format Font Underline OK

This is the text.
2

1
 TEST ORACLE
Test Case

Oracle Run-time
information from
Information executing GUI
Generator
Expected GUI State

Oracle Actual Execution

Procedure GUI State Monitor
Oracle
Verdict

• Oracle Information Generator - Expected state

• Execution Monitor - Actual state
• Oracle Procedure compares actual and expected state 3

GUI TEST ORACLE

• Components
• Oracle Information – Expected State
• Oracle Procedure – Compare
• Oracle Information Generator
• Manual
• Model Based – From Specifications
• Execution Based
? Screen Scraping
? Execution Extraction <==

• Oracle Procedure
• Actual Output <==> Oracle Information
• May be a set of rules for checking actual output
• May be an equality check

2
 CHALLENGES OF THIS WORK
• Design GUI representation that can be tuned
to create multiple test oracles for GUIs
• Design multiple oracle information for GUIs
• Develop compatible oracle procedure for
GUIs
• Develop metrics for comparing different types
of GUI test oracles

OUTLINE

• Related Work
• GUI State
• Oracle Information
• Oracle Procedure
• Combining Oracle
Information and Procedure
• Experiments
• Tool support
• Fault seeding
• Results

3
 RELATED WORK
• TAOS: Testing and Analysis with Oracle
Support, D.J. Richardson, ISSTA 1994.
• Manual
• Specification based
• TOBAC: A Test Case Browser for Testing
Object-Oriented Software, E. Siepman and
A.R. Newton, ISSTA 1994.
• Suggests 7 different oracles for object oriented
programs

OUTLINE
• Related Work
• GUI State
• Oracle Information
• Oracle Procedure
• Combining Oracle
Information and Procedure
• Experiments
• Tool support
• Fault seeding
• Results

4
 TEST CASE AND GUI STATE

T = e1 e2 … ei … en-1 en

S0 S1 S2 … Si-1 Si … Sn-1 Sn
INITIAL STATE

T = GUI test case of length n

ei = ith GUI event of test case
S0 = Initial State of the GUI
• A GUI test case consists of GUI events
• The state of the GUI changes as T is executed
• Event ei is executed on state Si-1 to yield state Si
9

GUI STATE
A GUI Window Properties of
Color = “Grey”
Height = 40
Width = 100
Text = “Cancel”

A WIDGET
State S = {(wi , pj , vk )}
Where SFIND = { (“Cancel”, “Color”, “Grey”),
w? W (“Cancel”, “Height”, 40),
p? P (“Cancel”, “Width”, “100”),
v? V (“Cancel”, “Text”, “Cancel”), …
}
• State of the entire GUI is the union of all window states
• Multiple types of oracle information can be created by selecting subset 10

5
 OUTLINE
• Related Work
• GUI State
• Oracle Information
• Oracle Procedure
• Combining Oracle
Information and Procedure
• Experiments
• Tool support
• Fault seeding
• Results

11

GUI TEST ORACLE

Test Case

Oracle Run-time
information from
Information executing GUI
Generator
Oracle Information

Oracle Actual Execution

Procedure State Monitor
Oracle
Verdict

• Oracle information for a test case is the expected state of the

GUI when the test case is executed
12

6
 ORACLE INFORMATION
• Given a Test Case T = < S0; e1, e2, …, en >
• S0 = State of GUI before executing T
• ei = ith event of test case
• Oracle Information OI = < S1 ,… Sn>
• Si = State of GUI after event e i is executed
• Si may be complete or partial state of GUI

By varying the information in Si, different Levels of

Oracle Information (LOI) can be created

13

WIDGET – LOI1
T = e1 e2 … Cancel … en-1 en

S0 S1 S2 … Si-1 Si … Sn-1 Sn

COMPLETE STATE OF GUI AFTER CANCEL

(wF,1, p 1, v1) (wX,1, p 1, v1) (wY,1, p 1, v1)
(wF,2, p 2, v1) (wX,2, p 2, v1) (wY,2, p 2, v1)
… … …
(wF,…, p r, v1) (wX,…, p t, v1) (wY,…, p x, v1)

(CANCEL, COLOR, GREY) (wX,…, p 1, v1) (wY,…, p 1, v1)

(CANCEL, HEIGHT, 40) (wX,…, p 2, v1) (wY,…, p 2, v1)
(CANCEL, WIDTH, 100) … …
(CANCEL, TEXT, CANCEL) (wX,…, p u, v1) (wY,…, p y, v1)

TRIPLE FOR (wF,…, p 1, v1) (wX,…, p 1, v1) (wY,…, p 1, v1)

(wF,…, p 2, v1) (wX,…, p 2, v1) (wY,…, p 2, v1)
CANCEL … … …
(wF,m, p s , v1) (wX,n, p v, v1) (wY,o, p z, v1)

• SAPP(t) = { (wi , pj , vk ) } Window F Window X Window Y

• After each event ei, select triples for the active widget ww,j
• ww,j is the widget on which event ei is executed
14

7
 ACTIVE WINDOW – LOI2
T = e1 e2 … match case … en-1 en

S0 S1 S2 … Si-1 Si … Sn-1 Sn

COMPLETE STATE OF GUI AFTER MATCH CASE

(wF,1, p 1, v1) (wX,1, p 1, v1) (wY,1, p 1, v1)
(wF,2, p 2, v1) (wX,2, p 2, v1) (wY,2, p 2, v1)
… … …
(wF,, p a, v1) (wX,…, p t, v1) (wY,…, p x, v1)

(wF,.., p 1, v1) (wX,…, p 1, v1) (wY,…, p 1, v1)

(wF,.., p 2, v1) (wX,…, p 2, v1) (wY,…, p 2, v1)
… … …
(wF,.., p b, v1) (wX,…, p u, v1) (wY,…, p y, v1)

TRIPLE FOR (wF,.., p 1, v1) (wX,…, p 1, v1) (wY,…, p 1, v1)

(wF,.., p 2, v1) (wX,…, p 2, v1) (wY,…, p 2, v1)
FIND … … …
WINDOW (wF,m, p c , v1) (wX,n, p v, v1) (wY,o, p z, v1)

Window F Window X Window Y

• SAPP(t) = { (wi , pj , vk )}
• After each event ei, select triples for the active window
• The active window is W, on which event ei is executed
15

VISIBLE WINDOWS - LOI3

isVisible = TRUE

T = e1 e2 … ei … en-1 en
isVisible = TRUE
isVisible = FALSE

S0 S1 S2 … Si-1 Si … Sn-1 Sn

• SAPP(t) = { (wi , pj , vk )}
• After each event ei, select triples for all the visible windows

16

8
 ALL WINDOWS – LOI4

T = e1 e2 … ei … en-1 en

S0 S1 S2 … Si-1 Si … Sn-1 Sn

• SAPP(t) = { (wi , pj , vk )}
• After each event ei, select all triples for all the windows
• All triples taken together is the complete state of the GUI

17

OUTLINE

• Related Work
• GUI State
• Oracle Information
• Oracle Procedure
• Combining Oracle
Information and Procedure
• Experiments
• Tool support
• Fault seeding
• Results
18

9
 ORACLE PROCEDURE
• At specified times, while executing a test case,
compares the actual state of the executing GUI with
the Oracle Information (OI), using some constraints.
AS = actual state OI = expected state

Level of Oracle
Procedure

Different Levels of Oracle Procedure (LOP) are created by

constraining
• the AS and OI used for comparing,
• the frequency of calling the Oracle Procedure
19

WIDGET – LOP1
OI = Oracle Information
(wX,1, p1, v1) (wY ,1, p1, v1) (wZ,1, p1, v1) (wX,1, p1, v1) (wY ,1, p1, v1)
(wX,2, p2, v1) (wY ,2, p2, v1) (wZ,2, p2, v1) (wX,2, p2, v1) (wY ,2, p2, v1)
… … … … …
(wX,…, pt , v1) (wY ,…, px , v1) (wZ,…, pa, v1) (wX,…, pt , v1) (wY ,…, px , v1)
(wX,…, p1, v1) (wY ,…, p1, v1) (wZ,…, p1, v1) (wX,…, p1, v1) (wY ,…, p1, v1)
(wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1) (wX,…, p2, v1) (wY ,…, p2, v1)
… … … … …
(wX,…, pu, v1) (wY ,…, py , v1) (wZ,…, pb, v1) (wX,…, pu, v1) (wY ,…, py , v1)
(wX,…, p1, v1) (wY ,…, p1, v1) (wz,…, p1, v1) (wX,…, p1, v1) (wY ,…, p1, v1)
(wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1) (wX,…, p2, v1) (wY ,…, p2, v1)
… … … LOP1 … …
(wX,n, pv , v1) (wY ,o, pz, v1) (wZ,o, pc , v1) (wX,n, pv , v1) (wY ,o, pz, v1)
Window X Window Y Window Z Window X Window Y
Verdict

• After each event ei, select triples from AS and OI for

the active widget ww,j
• ww,j is the widget on which event ei is executed 20

10
 ACTIVE WINDOW – LOP2
AS = Actual State OI = Oracle Information
(wX,1, p1, v1) (wY ,1, p1, v1) (wZ,1, p1, v1) (wX,1, p1, v1) (wY ,1, p1, v1)
(wX,2, p2, v1) (wY ,2, p2, v1) (wZ,2, p2, v1) (wX,2, p2, v1) (wY ,2, p2, v1)
… … … … …
(wX,…, pt , v1) (wY ,…, px , v1) (wZ,…, pa, v1) (wX,…, pt , v1) (wY ,…, px , v1)
(wX,…, p1, v1) (wY ,…, p1, v1) (wZ,…, p1, v1) (wX,…, p1, v1) (wY ,…, p1, v1)
(wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1) (wX,…, p2, v1) (wY ,…, p2, v1)
… … … … …
(wX,…, pu, v1) (wY ,…, py , v1) (wZ,…, pb, v1) (wX,…, pu, v1) (wY ,…, py , v1)
(wX,…, p1, v1) (wY ,…, p1, v1) (wz,…, p1, v1) (wX,…, p1, v1) (wY ,…, p1, v1)
(wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1) (wX,…, p2, v1) (wY ,…, p2, v1)
… … … LOP2 … …
(wX,n, pv , v1) (wY ,o, pz, v1) (wZ,m , pc , v1) (wX,n, pv , v1) (wY ,o, pz, v1)
Window X Window Y Window Z Window X Window Y
Verdict
•After each event ei, select triples, from AS and OI, for the active
window
• Active Window is W on which event ei is executed
21

VISIBLE WINDOWS - LOP3

AS = Actual State OI = Oracle Information
(w1, p1, v1) (w1, p1, v1) (w1, p1, v1)
(w1, p2, v1) (w1, p2, v1) (w1, p2, v1) (wX,1, p1, v1) (wY ,1, p1, v1)
… … … (wX,2, p2, v1) (wY ,2, p2, v1)
(w1, pn, v1) (w1, pn, v1) (w1, pn, v1) … …
(wX,…, pt , v1) (wY ,…, px , v1)
(wi, p1, v1) (wi, p1, v1) (wi, p1, v1)
(wi, p2, v1) (wi, p2, v1) (wi, p2, v1) (wX,…, p1, v1) (wY ,…, p1, v1)
… … … (wX,…, p2, v1) (wY ,…, p2, v1)
(wi, pn, v1) (wi, pn, v1) (wi, pn, v1) … …
(wX,…, pu, v1) (wY ,…, py , v1)
(wm , p1, v1) (wm , p1, v1) (wm , p1, v1)
(wm , p2, v1) (wm , p2, v1) (wm , p2, v1) (wX,…, p1, v1) (wY ,…, p1, v1)
… … … LOP3 (wX,…, p2, v1) (wY ,…, p2, v1)
(wm , pn, v1) (wn, po, v1) (wo, pm , v1) … …
(wX,n, pv , v1) (wY ,o, pz, v1)
Window W Window X Window Y
Visible Visible Invisible Window X Window Y
Verdict

• After each event ei, select triples, from AS and OI, for all visible
windows

22

11
 ALL WINDOWS – LOP4
AS = Actual State OI = Oracle Information
(wX,1, p1, v1) (wY ,1, p1, v1) (wZ,1, p1, v1) (wX,1, p1, v1) (wY ,1, p1, v1) (wZ,1, p1, v1)
(wX,2, p2, v1) (wY ,2, p2, v1) (wZ,2, p2, v1) (wX,2, p2, v1) (wY ,2, p2, v1) (wZ,2, p2, v1)
… … … … … …
(wX,…, pt , v1) (wY ,…, px , v1) (wZ,…, pa, v1) (wX,…, pt , v1) (wY ,…, px , v1) (wZ,…, pa, v1)

(wX,…, p1, v1) (wY ,…, p1, v1) (wZ,…, p1, v1) (wX,…, p1, v1) (wY ,…, p1, v1) (wZ,…, p1, v1)
(wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1) (wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1)
… … … … … …
(wX,…, pu, v1) (wY ,…, py , v1) (wZ,…, pb, v1) (wX,…, pu, v1) (wY ,…, py , v1) (wZ,…, pb, v1)

(wX,…, p1, v1) (wY ,…, p1, v1) (wz,…, p1, v1) (wX,…, p1, v1) (wY ,…, p1, v1) (wz,…, p1, v1)
(wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1) (wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1)
… … … LOP4
… … …
(wX,n, pv , v1) (wY ,o, pz, v1) (wZ,o, pc , v1) (wX,n, pv , v1) (wY ,o, pz, v1) (wZ,o, pc , v1)

Window X Window Y Window Z Window X Window Y Window Z

Verdict

• After each event ei, select all triples from AS and OI for all
the windows
23

FINAL STATE – LOP5

AS = Actual State OI = Oracle Information
(wX,1, p1, v1) (wY ,1, p1, v1) (wZ,1, p1, v1) (w1, p1, v1) (w1, p1, v1) (w1, p1, v1)
(wX,2, p2, v1) (wY ,2, p2, v1) (wZ,2, p2, v1) (w1, p2, v1) (w1, p2, v1) (w1, p2, v1)
… … … … … …
(wX,…, pt , v1) (wY ,…, px , v1) (wZ,…, pa, v1) (w1, pn, v1) (w1, pn, v1) (w1, pn, v1)

(wX,…, p1, v1) (wY ,…, p1, v1) (wZ,…, p1, v1) (wi, p1, v1) (wi, p1, v1) (wi, p1, v1)
(wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1) (wi, p2, v1) (wi, p2, v1) (wi, p2, v1)
… … … … … …
(wX,…, pu, v1) (wY ,…, py , v1) (wZ,…, pb, v1) (wi, pn, v1) (wi, pn, v1) (wi, pn, v1)

(wX,…, p1, v1) (wY ,…, p1, v1) (wz,…, p1, v1) (wm , p1, v1) (wm , p1, v1) (wm , p1, v1)
(wX,…, p2, v1) (wY ,…, p2, v1) (wZ,…, p2, v1) (wm , p2, v1) (wm , p2, v1) (wm , p2, v1)
… … … LOP5 … … …
(wX,n, pv , v1) (wY ,o, pz, v1) (wZ,o, pc , v1) (wm , pn, v1) (wm , pn, v1) (wm , pn, v1)

Window X Window Y Window Z Window W Window X Window Y

Visible Invisible
Verdict Visible

• After last event en, select all triples from AS and OI

24

12
 OUTLINE
• Related Work
• GUI State
• Oracle Information
• Oracle Procedure
• Combining Oracle
Information and Procedure
• Experiments
• Tool support
• Fault seeding
• Results

25

COMBINING LOI AND LOP

AS = Actual State OI = Oracle Information
(w1, p1, v1) (w1, p1, v1) (w1, p1, v1) (w1, p1, v1) (w1, p1, v1) (w1, p1, v1)
(w1, p2, v1) (w1, p2, v1) (w1, p2, v1) (w1, p2, v1) (w1, p2, v1) (w1, p2, v1)
… … … … … …
(w1, pn, v1) (w1, pn, v1) (w1, pn, v1) (w1, pn, v1) (w1, pn, v1) (w1, pn, v1)

(wi, p1, v1) (wi, p1, v1) (wi, p1, v1) (wii, p11, v11) (wi, p1, v1) (wi, p1, v1)
(wi, p2, v1) (wi, p2, v1) (wi, p2, v1) (w (wi, p2, v1) (wi, p2, v1)
… … … (wii,, pp22,, vv11))
… … …
(wi, pn, v1) (wi, pn, v1) (wi, pn, v1) (wi, p…n, v1) (wi, pn, v1) (wi, pn, v1)

(wm , p1, v1) (wm , p1, v1) (wm , p1, v1) (w , p , v )

(wmi , pn1, v11) (wm , p1, v1) (wm , p1, v1)
(wm , p2, v1) (wm , p2, v1) (wm , p2, v1) (wm , p2, v1) (wm , p2, v1) (wm , p2, v1)
… … … LOP1 … … …
(wm , pn, v1) (wm , pn, v1) (wm , pn, v1) (wm , pn, v1) (wm , pn, v1) (wm , pn, v1)

Window W Window X Window Y Window W Window X Window Y

Visible Visible Invisible Visible Invisible
Verdict Visible

• LOP1 may be used with LOI1, LOI2, LOI3, LOI4

26

13
 TYPES OF ORACLES
• An oracle procedure may use less than available
oracle information (OI)
• The OI can be generated before running the test case
• Depending on resources available at OI collection,
different OI is created
• Depending on resources available at test case
execution time, the oracle procedure may use a subset
of available OI
• The 4 LOI and 5 LOP can be combined into 11 test
oracles

27

COMBINING LOI AND LOP

LOP1 LOP2 LOP3 LOP4 LOP5

LOI1 X

LOI2 X X

LOI3 X X X

LOI4 X X X X X

• For the diagonal elements, all the Oracle Information

is used by the Oracle Procedure
• For the sub-diagonal elements, the Oracle Procedure
uses less than available Oracle Information 28

14
 OUTLINE

• Related Work
• GUI State
• Oracle Information
• Oracle Procedure
• Combining Oracle
Information and Procedure
• Experiments
• Tool support
• Fault seeding
• Results
29

QUESTIONS

• What is the cost incurred in using these

oracles ?
• Do different types of oracles detect different
number of faults for a given number of test
cases ?
• Length of test case versus oracle types.

30

15
 APPROACH

• 4 applications
• Seeded 100 faults to create 100 mutants
• Generated 600 test cases for each application
• Reported results of 4 x 100 x 600 = 240,000
test case runs

31

PROCESS
• Generate test cases
• Run test case on correct application
• Collect oracle information (LOI1-LOI4)
• Run test case on mutants
• Compare the actual state with stored oracle
information using different levels of oracle
procedure (LOP1-LOP5)

32

16
 SUBJECT APPLICATIONS
• Subject Applications
• TerpOffice – 4 Java applications
• Developed by students of undergraduate software
engineering (CMSC 435)

LOC Classes Windows

TerpWord 1,747 9 8
TerpPresent 4,769 4 5
TerpPaint 9,287 42 8
TerpSpreadSheet 9,964 25 6
TOTAL 25,767 80 27

• Fairly large programs

33

MUTANTS
• Seeded single fault in original code to create
one mutant
• Single fault avoids fault interaction
• Easier to identify a fault when a mutant it
killed
• Seeded 100 faults for each application
• Total 400 mutants

34

17
 KILL MUTANTS
• A mutant is ‘killed’ when a test case
distinguishes it from the original application
• Execute a test case on each mutant
• Compare actual output, AS, of mutant with
expected output from OI
• Use different types of oracle for comparing
• Using different oracles, a mutant may or may not
be killed for a test case
• Execute 600 test cases for each application

35

TOOL SUPPORT
• Manual process
• Executing a test case with 20 events takes
minute
• Observing and recording Oracle
Information is time consuming
• Comparing actual state with expected state
is time consuming
• Automated the entire process

36

18
 TOOL SUPPORT
GUI
APPLICATION
REPRESENTATION

ORACLE INFORMATION 100

GENERATOR MUTANTS

TEST CASE GENERATOR

TEST CASE EXECUTOR,
ORACLE PROCEDURE

GUI TEST
TEST CASES ORACLE INFORMATION
REPORT

TESTING ARTIFACT Intermediate data generated by tool

TESTING TOOL Testing tool from GUITAR 37

OUTLINE
• Related Work
• GUI State
• Oracle Information
• Oracle Procedure
• Combining Oracle
Information and Procedure
• Experiments
• Tool support
• Fault seeding
• Results

38

19
 ORACLE INFORMATION
GENERATION TIME
14
12
LOI1
Time (hours)

10
LOI2
8
LOI3
6 LOI4
4 Final State

2
0
TerpPresent TerpWord TerpSpreadSheet TerpPaint

Final state required least time because only one state was queried

39

ORACLE STORAGE SPACE

250

200
Storage Size (MB)

LOI1
150 LOI2
LOI3
100 LOI4
Final state

50

0
TerpPresent TerpWord TerpSpreadSheet TerpPaint

• Space increases from LOI1 to LOI4

• Storing the final state requires less space than LOI2-LOI4
40

20
 COMPARISON TIME

700

600

500
LOP1
Time (hours)

400 LOP2
LOP3
300 LOP4
LOP5
200

100

0
TerpPresent TerpWord TerpSpreadSheet TerpPaint

• Comparison time for LOP5 is least for all applications

• Only the final state was obtained
41

TYPES OF ORACLES
LOP1 LOP2 LOP3 LOP4 LOP5

LOI1 X L1

LOI2 X X L2

LOI3 X X X L3

LOI4 X X X X L4 X L5

• Diagonal entries labeled as L1…L5

• Faults detected for sub-diagonal entries is same as diagonal entry
• The LOP uses subset of LOI
42

21
 FRACTION OF FAULTS
100

80
% of Faults Detected

L1
60 L2
L3
40 L4
L5

20

0
TerpPresent TerpWord TerpSpreadSheet TerpPaint

LOP LOP LOP LOP LOP

1 2 3 4 5
LOI1 X L1

L3, L4, L5 found more faults than L1 LOI2 X X L2

and L2 LOI3

LOI4
X

X
X

X
X

X
L3

X L4 X L5

43

FIND FAULTS EARLY

6000
Number of Test Cases Detecting Faults

5000

4000 L1
L2
3000
L3
2000 L4

1000

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Test Case Length

• Defects found earlier with L3, L4

44

22
 NUMBER OF FAULTS DETECTED BY A
TEST CASE
2000

1500
Number of Test cases

2000
1000 1800
1600

Number of Test cases

500 1400
1200
0 1000
1 2 800
3 4 L5
5 6
7 8 600
9 10 L3
11 12 400
13 14
Number of Faults 15 16 L1
17 18 200
19 20
0
2 1
5 4 3
L1 9 8 7 6
10
12 11
L4 14 13 Number of Faults
17 16 15
18
20 19

• Larger number of test cases find faults using L3, L4, L5

45

LESSONS LEARNED
• Different oracles have different cost and fault
detection ability
• Use of “Final State” is cost effective
• Less storage and generation time
• Found almost as many fault as “Visible Window”
and “All Windows”
• If short test cases are available
• Use an “Visible Windows” and “All Windows”
• “Final State” may miss faults that are masked,
by the time the last event is executed

46

23
 CONTRIBUTIONS
• Defined GUI state, so that is can be tuned to
create multiple test oracles
• Developed GUI test oracles
• Oracle information
• Oracle procedure
• Developed tools for experiment
• Designed and conducted an experiment
• Developed guidelines for GUI testing

47

BROADER IMPACTS
• Applied multiple test oracles for effective
regression testing (ICSM 2003)
• Developed reverse engineering tool for GUIs
(WCRE 2003)
• Compared multiple test oracles for GUI testing
(ASE 2003)
• Collaborated with Hughes Network Systems
on developing testing tools

48

24
 FUTURE WORK
• Metrics to evaluate and compare test oracles
• Use Windows applications as subjects
• Generate multiple types of test cases and
observe the effect of different oracles

49

25