Building a user interface test automationframework

Izzat Alsmadi and Kenneth MagelDepartment of Computer Science North Dakota State UniversityFargo, ND 58105{ izzat.alsmadi, kenneth.magel}@ndsu.edu

Abstract

This article suggests a fully test automation tool with the least amount of timeand decisions required from testers. Using an automated tool should not beoffset by testers spending a good amount of time for setup and guidance.The proposed framework for user interface test automation uses the informationstored in the application assembly to build the tested model. The GUI model is parsed from the application under test at run time. The GUI model is built as anXML tree. Test cases are then generated from the XML tree using different proposed algorithms. Some techniques for test case prioritization, critical pathtesting, execution and verification are suggested. The goal is to do all thoseactivities dynamically as part of a GUI test automation framework.

General Terms

Software testing, user interface testing and testautomation.

Keywords

Software test automation, test case generation, Testexecution, GUI testing, GUI modeling, andverification.

1. Introduction

Software testing is an important stage in software projects. It is one of the most expensive stages thatusually stops after using the available resources (andnot completed). Test automation takes more time inthe initial stages. Eventually, this is expected toreduce the cost of testing through reducing therequired time and resources. GUI is increasinglytaking a larger portion of the overall program’s sizeas newer programs are more interactive with the user inputs. and GUI testing is taking a major rule in thewhole project’s validation.GUI test automation is a major challenge for testautomation activities. Most of the current GUI testautomation tools are partially automated and requirethe involvement of users or testers in several stages.Test automation tools are still complex andexpensive. They don’t fully replace testers. They can be usually used to re-execute those repeated tasks.Companies consider taking the decision to buy, or not, a GUI test automation tool as a tough decisionsince they don’t know how much time it will requireupfront for setup. They also don’t know how much of test automation tasks can be fully or partiallyautomated. We all agree that complete coverage intest automation is impossible as we also agree thatcomplete coverage in testing is impossible too. No matter what software methodology a companyfollows, software testing is going to play a major rolein terms of project success or resource allocation.Automated test refers to testing by a computer (nohuman tester). We choose the option of automatingthe GUI testing when tests are repeated several times[1]. Software test automation tasks include selectingand generating the test cases, building the test oracle(test cases with expected results), executing, re-executing, reporting and validating the results.GUIs manage controls. Controls are reusable objectswith which users can interact. The term "control" isused as a generic term for any graphical object or widget that an application may produce. Controlshave common elements that should be considered before writing a program that interacts with a GUI[2].

•

Controls are categorized according to Windowclasses. Each control belongs to a window class(making it possible to search them by class).

•

Controls have an organizational hierarchy; everyGUI has at least one root control, and every controlmay have child controls. Controls and forms make atree. We can generate a valid test case by selectingseveral controls that are children of each other.

•

Controls have different ways to be located or identified at run time. They can be identified by their names, parents, unique location or by the windowhandle associated to them at run time.We present a GUI testing framework that does notrequire the involvement of testers throughout thedifferent stages. The tests do not need to be revisitedduring any change in the GUI structure. Since theapplication uses reflection to get the actual model atrun time, those changes are included in the currenttest.Generating test cases can happen from requirements,design or the actual GUI implementation. Although itis expected that those three should be consistent andrelated, yet they have different levels of abstraction.Requirements and design are usually of a high levelof abstraction to generate from them the test cases.On the other hand the task of generating test casesfrom the GUI implementation model is delayed untilGUI is implemented, which is usually occurred in thelate implementation. We should not have any problems in delaying GUI testing giving the fact thata tool will automate the generation and execution processes.A tool is built in C# that uses reflection to serializethe GUI control components. Certain control properties are selected to be serialized. These properties are relevant to the user interface testing.The application then uses the XML file that is createdto build the GUI tree or the event flow graph. Testcases are then generated using the GUI XML file.Generating the test cases takes into consideration thetree structure to select the test cases that cover unique branches.In test execution and verification, test cases areexecuted on the application and compared with theoriginal test suite. The advantages of the objectoriented approach over the widely usedcapture/replay back one, is in the fact that the modelis generated at run time which makes it represents thecurrent state of the GUI model. In record/play back cases, we have to retest the application in case of anychange in the functionalities or the GUI of the program. Once utilized, this object oriented approachis expected to be less expensive as it does not requireusers to manually test the application or makedecision for pre- and post-conditions.

2. Related work

There are several GUI test automation tools availablein the market. Some of the larger vendors for suchtools includes: IBM Rational, Mercury, and Segue.Trying to build a GUI test automation tool is liketrying to make a new operating system in such thatthe available resources for the existed ones are hardto compete with. However, GUI test automationdidn’t reach a mature level were those tools can beimplemented in all scenarios with no problems. User Interfaces evolves rapidly with more rich and newcomponents. The automation processes in general useartificial intelligence algorithms to simulate user activities. The richness and complexity of this spacemake it possible for new research ideas to competewith long time existed test automation tools in theindustry.There are several research papers about GUI testingusing the data model, [3] [4] [5] [7] [8] [9] [10]. Theoverall goals and approach for this work is verysimilar to their goals. The GUI testing framework described, as a GUI test automation structure, isgeneric and should be applied to any testing or GUItesting model. Since research projects in general havelimited resources, each paper discusses a specificarea of GUI test automation. Without having allcomponents, it is very hard to use such ideas in theindustry. There is a need for using universalstandards for the output formats in all testingactivities. If each tool is producing severalintermediate outputs (generated test cases, recordedscripts, log files, etc), it will be impossible to usethose tools or ideas in other research projects. Some papers follow a complex procedure in test casegeneration and do not consider any state reductions.Assuming that changing any property in any GUIobject changes the GUI state is an assumption thatgenerated a very large number of possible states for even small applications. State reduction techniquesare considered here to improve the effectiveness of the track. We intended to follow the same GUItesting framework for our future work and expect theoverall results to be more practical and easier toapply on actual projects.The second category[13][6] is related to semi testautomation using some capture/reply tools likeWinRunner, QuickTest pro, Segui silk, QARun,Rational Robot, JFCUnit, Abbot and Pounder tocreates unit tests for the AUT. Capture/reply toolshave been existed and used for years. This may makethem currently more reliable and practical as theyhave been tested and improved through severalgenerations and improvements. There are several problems and issues in using record/play back tools[11]. The need to reapply all test cases when the GUIchanges, the complexity in editing the scripts code,and the lack of error handlings are some examples of those issues. The reuse of test oracles is not veryuseful in the case of using a capture/replay tool [12].We expect future software projects to be more GUIcomplicated that may make the test automation datamodel more practical. Many researches and

improvements need to be done for the suggested datamodel to be more practical and usable.

3. Goals and approaches

Our approach uses two constraints: The tool shoulddevelop all test automation activities without or withthe least user involvement. The second constraint isthat those dynamically generated test case should provide a value for test adequacy and shouldrepresent actual test case scenarios.Since the test cases will be dynamically generatedwith no need for a user to define pre- or post-conditions, algorithms are developed for test casegeneration.One important aspect of GUI’s that helps in statereduction is the fact that the GUI is hierarchical bynature. For example, in MS Notepad, to reach theevent “Print”, the event “File” should be executedfirst.The GUI is analyzed and its hierarchical tree of controls is extracted using reflection. The decision touse the XML file as a saving location for the GUImodel is in the fact that XML is a universal formatand it supports hierarchy. This hierarchy can beextracted easily from the XML file. We used theinformation about the parent of each control in theassembly to build the GUI tree. Figure1 is a screenshot from the XML output file extracted from asimple Notepad application.<Root>GUI-Forms</Root><Root>Form1</Root><Parent-Form>Form1</Parent-Form><Name>Form1</Name><Control-Level>0</Control-Level><TextBox><Parent-Form>Form1</Parent-Form><Name>textBox1</Name><Control-Level>1</Control-Level><ControlUnit>0</ControlUnit><LocationX>0</LocationX><LocationY>24</LocationY><Forecolor>Color [DarkBlue]</Forecolor><BackColor>Color [Linen]</BackColor><Enabled>True</Enabled><Visible>False</Visible></TextBox><MenuItem><Name>System.Windows.Forms.MenuItem,Items.Count: 4, Text: File</Name><Control-Level>1</Control-Level><ControlUnit>0</ControlUnit><Parent>System.Windows.Forms.MainMenu,Items.Count: 3</Parent><Text>File</Text><Visible>True</Visible><Enabled>True</Enabled><ShortCut>None</ShortCut></MenuItem>Figure1: a screen shot from a generated XML file.The above information about the controls and their properties are extracted directly from the assembly.We added two more properties; control level andcontrol unit for each control. Those properties areused along with the parent property to locate eachcontrol during test generation, selection andexecution.The user has the ability to select certain controls fromthe tree and gives them more than the typical equallydistributed weight. This will have an effect on testcase selection algorithms. For example, a control thatis selected manually by the user will be selectedwhenever it is possible or valid to do so.Rather than defining the GUI state change as achange in any control or control property [2], which produces a large amount of possible test cases, wedefine the GUI state as the state that is represented bythe generated XML file from that GUI. If any of the parent-child relations in the GUI is changed, or any property of those parsed controls is changed, thenthat is considered a GUI state change.Some abstraction is used removing those propertiesthat less relevant or important to the GUI state toreduce the large number of possible states. Themanually selected properties are: the control name, parent name, text, locationX (horizontal location),locationY (vertical location), text, or caption,forecolor, backcolor, enabled, visible, control unitand control level.In the process of developing test generationtechniques, several test generation algorithms aredeveloped. All algorithms check for a valid selectionof a tree edge. For example, using Notepad AUT, if the current control is “File”, then a test algorithmmay select randomly a valid next control from thechildren of the File control ( e.g. Save, SaveAs,Open, Exit, Close, Print). In another algorithm, we processed the selected test scenarios to ensure that notest scenario will be selected twice in a test suite.Figure2 is a sample output generated from one of thetest generation algorithms. In the algorithm, each testcase starts from the root or the main entry “NotepadMain”, and then selects two or more controlsrandomly from the tree. The algorithm verifies thatthe current test case is not existed in the alreadygenerated test cases.1,NOTEPADMAIN,SAVE,SAVEFILEBUTTON1,,,2,NOTEPADMAIN,EDIT,FIND,TABCONTROL1,TABFIND,