You are on page 1of 21

TestComplete

by SMARTBEAR
2
Virtual machines or Virtual PC’s are software like VMware that
emulates real computers. By using virtual machines in your QA
testing, you can emulate different computers with different OS’s
on a single physical computer or create a whole virtual lab, with
several differently configured virtual machines. These virtual com-
puters work independently from each other and you can launch
two or more virtual platforms simultaneously on one computer,
saving the cost of having to purchase more hardware just to run
your QA tests.
TestComplete
by SMARTBEAR
3
Automated Testing with Virtual Machines ................................................. 4
An Affordable, Stable, System Confguration for QA Testing ............................ 5
Virtual Testing Labs on a Single Computer ....................................................... 5
Secure QA Testing in a Virtual Lab with its Own Network ................................. 6
Locking Computers in Your Virtual Lab ............................................................. 6
Automating Tests on a Virtual Machine ............................................................. 7
Automating Tests Using TestComplete ...................................................... 7
Automating Tests Using Automated Build Studio ....................................... 8
Running Automated Tests in Virtual Labs with TestComplete ........ 12
Preparing Virtual Labs for Automated Testing ................................................. 12
Ìnstalling an Automated Testing Tool ............................................................... 13
Controlling Automated Tests in Virtual Labs ................................................... 13
Virtual machines residing on a local computer ........................................ 14
Virtual machines residing on a remote computer ..................................... 14
Organizing the application's distributed testing ............................................... 15
Automated Testing in a Virtual Lab - Scheme ................................................. 16
Determining the Type of Virtual Machine in Automated Tests ......................... 17
Distributed Testing in Virtual Labs with TestComplete ...................... 18
Testing Web Applications in a Virtual Lab with TestComplete ........ 19
Virtual Machine Limitations .......................................................................... 20
Conclusion ............................................................................................................ 21
Table of Contents
TestComplete
by SMARTBEAR
4
Using virtual machines to create virtual testing labs is a common soft-
ware testing practice. Ìt allows QA teams to save money by running a
number of virtual machines on fewer computers, or to take advantage
of resources available in the Cloud. However, the effciency of testing
in virtual labs increases when you automate the testing process with an
automated testing tool like TestComplete.
TestComplete is a comprehensive automated testing tool with extended
features for testing applications running on virtual machines and it sup-
ports any kind of QA testing methodology: unit testing, regression test-
ing, GUÌ testing, Web testing, distributed testing and many others.
Applications running on a virtual machine behave as if they were run-
ning on their own physical system. This is very useful for testing applica-
tions and allows testers to test their application under various platforms
using a single computer. Ìt can also be helpful for testing web applica-
tions since you can simultaneously test your web application in multiple
browsers running independently from each other under different virtual
machines, again, without the cost of purchasing more hardware for your
testing needs.
Testing applications on virtual machines can have different purposes.
You can use it for distributed testing of client-server applications, func-
tional testing or in regression testing. But, no matter which kind of QA
testing you use, it will be more effective if automated and any kind of
testing in virtual labs can be easily automated with TestComplete.
To run your tests on virtual machines using TestComplete, you can
buy the Floating User license of TestComplete, activate it on a physi-
cal machine and then install TestComplete on virtual machines on your
network and use it to run the tests. For more information, see Using
TestComplete on Virtual Machines.
Besides virtual machines, you can also use cloud computers for applica-
Automated Testing with Virtual Machines
TestComplete
by SMARTBEAR
5
tion testing. For more information on running automated tests on cloud
computers, see Cloud Testing With TestComplete.
An Affordable, Stable, System Configuration for QA Testing
A computer system is a dynamic system and operating system confgu-
rations continuously change. Ìnstalling or updating software and hard-
ware drivers happens frequently and installing different versions of an
application affects the system's internal structure, the registry and other
crucial components that persuade testing results. While buying multiple
computers to support various platforms is an option for some, it is often
too expensive for most testing labs.
Since computer systems are ever changing, it causes a problem for
QA testing. We cannot fully understand testing results when testing is
performed on different confgurations. Ìn other words, if our application
does not work properly, it is more diffcult to determine the reason in a
constantly changing system. Another problem is that it is diffcult to test
different application builds installed on a single operating system. Ìt can
cause conficts and may even provoke system crashes.
Fortunately, virtual machines are a workaround for these problems and
are a lot more cost effective. Once you have created and confgured a
virtual machine or a full lab of virtual machines, you will have a stable
system confguration, which is very important when testing applications.
Benefits of Virtual Testing Labs on a Single Computer
Virtual machines can work independently from each other on a single
physical computer, creating a virtual testing lab on one computer. This is
a huge beneft which is often used in QA testing.
For example, you can use one computer to test your application under
Windows XP, Windows Vista and Windows 7. Ìn virtual labs, there is
no need for three different computers to execute tests simultaneously;
you can just sequentially execute tests on a single computer with three
different virtual confgurations. However, you may need a more powerful
TestComplete
by SMARTBEAR
6
computer to run several virtual machines on one computer at once, but
this is often cheaper than buying three physical computers.
Create Complex Test Environments
Sometimes testing applications is an unpredictable process ÷ a tested
application may hang the operating system, provoke a system crash
and endanger the corporate network's security. To isolate your testing
lab, you can join your virtual labs into a subnet within your network and
then isolate it from the rest of the computers in your company. You get a
virtual lab network fully isolated from your company's computers that will
increase the security level of your testing process.
Ìsolating your virtual lab on its own network does not determine the
type of software testing you perform; you can still perform distributed
testing, GUÌ testing, web testing and any other kind of testing. You can
also remotely launch automated tests on virtual machines and use any
automated testing technique provided by TestComplete.
Locking Computers in Your Virtual Lab
QA testing teams may want to execute their automated software tests
on locked computers when running tests overnight. Software tests that
don't interact with the GUÌ can be run on locked computers. GUÌ tests
or functional tests that simulate user actions cannot be run on locked
computers, since the user session is frozen (no windows can be shown)
and the automated testing tool cannot simulate user actions.
Virtual machine software allows you to run GUÌ and functional tests on
locked computers with TestComplete. To do this:
Ìnstall TestComplete and your application on a virtual machine.
Also, copy all fles that your application and TestComplete project
may require for testing.
Lock the virtual machine's host computer.
Log onto the virtual machine from any other computer.
TestComplete
by SMARTBEAR
7
Start executing tests.
Note that you can easily automate this process using Automated Build
Studio. For more information, see the Automating Tests Using Automat-
ed Build Studio section below.
TestComplete will run tests on the virtual machine despite the host be-
ing locked. However, if you disconnect from the virtual machine while
running GUÌ tests, the automated tests will fail.
Automating Tests on a Virtual Machine
There are various approaches that let you automate the test runs on the
virtual machines. The sections below describe two of them.
Manage Virtual Machines with Custom Scripts
Usually virtual machine software provides a special COM APÌ that you
can use to manage virtual computers. Ìt is easy to work with COM
objects from TestComplete, so you can use this COM APÌ to automate
testing over virtual machines directly from your virtual testing lab.
The following code sample (VBScript) demonstrates how to automate
the running and stopping of a VMware virtual machine directly from
an automated test script. We use a .NET wrapper for VÌX COM APÌ to
power on and then to power off a virtual PC. This script routine works
with the VMware Workstation installed on the computer where the rou-
tine is launched.
The dotNET object is a special TestComplete object that you can use to
call routines and methods that reside in any .NET assembly from your
automated tests.
TestComplete
by SMARTBEAR
8
VBScript:
Manage Virtual Machines with Automated Build Studio
Besides using COM to automate tests from TestComplete, you can also
use Automated Build Studio for managing virtual machines. Using Auto-
mated Build Studio is the easiest way to automate actions over virtual
platforms. Ìt contains a lot of high-level operations intended for working
with different VM software. Also, it is not necessary to write scripts or
perform other specifc tasks. Ìn other words, creating a sample macro
does not require a user to have any programming background.
One more advantage is that Automated Build Studio lets you create
macros in the visual editor:
Figure 1. The operations of Automated Build Studio.
TestComplete
by SMARTBEAR
9
Also, it is really easy to specify operation parameters as you do this in-
tuitively via special wizards and dialogs. For instance, to reboot a virtual
machine from an Automated Build Studio macro you can just add the
appropriate operation to the macro and specify all the required param-
eters in the enduing dialog. Then, for instance, you can schedule the
macro execution to run your virtual machine at a specifc time.
So, Automated Build Studio macro can start the virtual machine, launch
TestComplete and then close the machine when the test run is over.
Below, there is a sample macro created for the purposes described
above:
Figure 2. The sample macro.
This macro runs the needed virtual machine using the Run Virtual
Machine (Virtual PC) operation, waits until the operating system of the
machine is loaded completely and then runs TestComplete as a process
using the Run Process (WMI) operation.
TestComplete
by SMARTBEAR
10
Ìt is easier to perform some of the tasks mentioned above via TestCom-
plete command line. For example, to launch TestComplete and make
it run the desired test automatically, use the /project:project_name /
test:test_name command-line argument. You can also make TestCom-
plete to close right after the test execution is over. For this purpose, use
the /e command-line key. For more information on running TestCom-
plete tests from a command line, see TestComplete Command Line.
After that, Automated Build Studio checks whether the TestComplete is
running. You can check the value of the variable that stores the results
of the Ìf Process Is Running (WMI) operation in the While Loop and
set a delay each time when this value equals to True.
When TestComplete stops running, Automated Build Studio closes the
virtual machine and stops the macro execution.
Automated Build Studio also includes special TestComplete and
TestExecute operations for running automated tests with TestComplete
or TestExecute from within your Build Studio macros. The operations'
properties specify the project, test item or individual script routine to be
executed. The operations wait until the testing is over and then save test
results to a fle or post them to the macro log.
Figure 3. TestComplete Operation.
TestComplete
by SMARTBEAR
11
The best practice is to use Automated Build Studio and TestComplete
together. This allows you to automate both virtual machines and auto-
mated tests in a virtual lab easily.
Once you have created automated tests with TestComplete, you should
confgure the needed virtual machines in your virtual lab. To prepare
virtual machines for testing, determine the following requirements:
Which testing platforms do you need?
Which VM software should you choose?
Which automated testing tool do Ì need to install on virtual
machines?
These requirements are important for software testing and you are free
to choose any platform and use TestComplete to test your application.
You also need to determine how you will work with the tested platforms:
Will virtual machines run on a single or on several computers?
Which utility should Ì use for controlling virtual machines?
For more information on preparing virtual machine for automated test-
ing, see Using TestComplete on Virtual Machines.
Preparing Virtual Labs for Automated Testing
As we mentioned above, you are free to choose almost any testing plat-
form and any VM software available on the market to use in your virtual
testing labs.
After you have installed the particular VM software, you can prepare
your virtual machines according to the following steps:
1. Create a new virtual machine. For information on how to do this,
see the VM software documentation.
2. Confgure the virtual hardware.
What You Need to Know Before You Start Testing
TestComplete
by SMARTBEAR
12
3. Ìnstall the needed operating system.
4. Ìnstall TestComplete or TestExecute (see Ìnstalling Automated Test-
ing Tool section).
Installing an Automated Testing Tool
There are several tools that can be used to execute automated tests on
virtual machines in your virtual lab. For example, you can install Test-
Complete or TestExecute on the virtual platforms and then run the test
on these platforms using the installed tool.
To run your tests on virtual machines using TestComplete, you can buy
the Floating User license of TestComplete, activate it on a physical
machine and then install TestComplete on a virtual machine on your net-
work. You can then launch TestComplete and execute your automated
tests in your virtual lab. Note that all TestComplete plug-ins required for
your tests to also be installed and enabled in TestComplete in order for
you to use them in your virtual testing lab.
As an alternative, you can create automated tests with TestComplete on
a real computer and then copy these tests to several virtual machines
in your virtual testing lab. Then you can install TestExecute with all re-
quired plug-ins and run your automated tests in your virtual lab. TestEx-
ecute is a resource-friendly utility that executes TestComplete tests and
logs its results on machines that do not have TestComplete.
For more information on installing TestComplete, refer to the TestCom-
plete 9 Ìnstallation Guide document that is supplied along with TestCom-
plete. Ìt describes how you install, modify and uninstall TestComplete in
details.
For more information on installing and activating TestComplete license
on the virtual machines, see Using TestComplete on Virtual Machines.
TestComplete
by SMARTBEAR
13
Controlling Automated Tests in Virtual Labs
To be able to launch automated tests on virtual machines in your virtual
testing lab, they should be running and you should have control over
them, whether direct or remote control. There are two possible ways to
work with virtual machines in your lab:
Virtual machines residing on a local computer
Virtual machines can be installed and run on a local computer, where
TestComplete or TestExecute is also installed. You can control differ-
ent virtual machines just by switching between the windows with virtual
computer desktops.
When running the GUÌ tests in a virtual machine console, do not mini-
mize the console window. When a virtual machine window is minimized,
the operating system does not draw the GUÌ so TestComplete cannot
perform user actions over the tested application.
As for other kinds of tests, like unit tests, you can minimize the virtual
machine console; however, do not shut down the machine until the
automated tests are fnished.
Virtual machines residing on a remote computer
You can access remote virtual machines via a special utility, for in-
stance, the Remote Desktop Connection tool embedded in Windows
XP or third-party utilities, VNC, DameWare and others. Ìn this case the
virtual machine must be preliminary prepared for controlling it remotely
and executing automated tests.
The Remote Desktop is a built-in Windows feature that lets you access
a Windows session running at one computer from another computer. Ìn
order to use Remote Desktop, the workstation that has TestComplete or
TestExecute must have special server-side software. This software may
be included in the operating system (like it is included in Windows XP),
or it can be a third-party software like VNC, DameWare or NetOp.
TestComplete
by SMARTBEAR
14
The Remote Desktop feature lets you perform a variety of tasks. For
example, you can use the Remote Desktop if you need to:
Observe the tested application's behavior on remote computers.
Run tests on remote computers.
Run tests on computers that are locked due to security purposes
(see Locking Computers in Your Virtual Lab).
View and analyze test results on one or several remote computers,
and save them to a local computer.
To run automated tests with Remote Desktop, follow these steps:
Ìf needed, lock the computer where TestComplete (TestExecute) is
installed.
Log on to the TestComplete (TestExecute) workstation via the
Remote Desktop Connection from another computer.
Start running tests with TestComplete (TestExecute).
TestComplete (TestExecute) will run tests in the Remote Desktop ses-
sion, the same way as if they were running from the remote computer
itself.
When running GUÌ tests via the Remote Desktop, it is not recommend-
ed that you minimize the Remote Desktop window; when the Remote
Desktop window is minimized, the operating system does not draw the
GUÌ so TestComplete will not be able to "see¨ the tested application's
interface and thus will fail to interact with it.
However, it may inconvenient to always keep the Remote Desktop
Window visible as it occupies some part of screen. To solve this prob-
lem, you can change the Registry key on the machine from which you
access the remote workstation. For more information on this approach,
see the TestComplete Tip: Running Tests in Minimized Remote Desktop
Windows blog.
TestComplete
by SMARTBEAR
15
Organizing the application’s distributed testing
Also, TestComplete lets you test the application that can work with
several clients simultaneously. For example, you can use this approach
for testing web sites, client-server or other applications. We call it
distributed testing and using it you can run different parts of your test on
different machines connected with each other through the network. One
more advantage of this technique is that you can easily coordinate and
synchronize the test parts running on different machines.
For more information on this testing type, see the Distributed Testing
With TestComplete article.
Automated Testing in a Virtual Lab - Scheme
Now that the virtual machine confgurations are ready in your virtual
testing lab and you know how to control the automated test execution,
you can start software testing. The following automated testing scheme
allows you to take full advantage of your QA testing in a virtual lab:
1. Create a copy of the virtual machines with the desired system con-
fguration. Only test with this copy from this point forward.
2. Run the virtual machine copy manually or, for example, use the
Run Virtual Machine (Virtual PC) operation to automatically do it
from Automated Build Studio macro.
3. Ìnstall the tested application.
4. Copy all automated tests to the virtual machine in your virtual test-
ing lab.
5. Run automated tests under the virtual machine. As we have men-
tioned earlier, you can do it manually or you can easily automate it
with special operations in your Automated Build Studio macro.
6. Transfer all test results to the physical computer, where they can be
analyzed.
TestComplete
by SMARTBEAR
16
7. Turn off the virtual machine copy, for example, you can do it using
the Close Virtual Machine (Virtual PC) Automated Build Studio's
operation.
8. Delete the virtual machine copy.
9. Perform steps 1-8 for each automated test in your virtual lab.
With this scheme, you can test different application builds on different
operating systems with different confgurations.
Determining the Type of Virtual Machine in Automated Tests
TestComplete provides special properties that determine whether
TestComplete is running under a virtual machine or a physical computer.
With TestComplete, if the Sys.OsÌnfo.VirtualPC property is True, this
indicates that TestComplete is running under a Virtual PC machine. Ìf
the Sys.OsÌnfo.VMWare property is True, TestComplete is running under
a VMWare machine.
Figure 4. Determining the Type of Virtual Machine.
TestComplete
by SMARTBEAR
17
You can also determine whether TestComplete is running under a
remote session via the Sys.OsÌnfo.RemoteSession property. Ìf this
property is True, TestComplete is running under a remote session, for
instance, a session launched via the Remote Desktop utility. Using this
property you can determine, for instance, whether a tester logged into
a virtual workstation via a remote control utility and execute all tests,
except GUÌ tests, on this virtual machine.
You can use any of the properties mentioned above to provide condi-
tional testing in your automated tests. For example, you can execute
certain automated tests on specifc virtual machines and other auto-
mated tests on physical machines, based on particular property values.
Also, you can execute only those tests that do not require an Ìnternet
connection on virtual machines, and so on.
Distributed testing means that a test consists of several parts and each
part is performed on a separate computer. This testing is often per-
formed with Client-Server type applications. With a virtual lab on one
computer, you can perform distributed testing of your Client-Server ap-
plications without the need for multiple
computers.
With TestComplete, you can automate
the whole distributed testing process.
Distributed tests can be organized as a
master and slave projects in TestCom-
plete, and then run in a virtual lab.
This image demonstrates how two
parts of a distributed test are executed
on a single computer. The master proj-
ect is running on the physical computer
and the slave project is running on a
virtual machine that runs on that same computer:
Distributed Testing in Virtual Labs with TestComplete
Figure 5. Distributed Testing in Virtual Labs.
TestComplete
by SMARTBEAR
18
Using virtual machines is becoming a good practice for testing different
web applications. For instance, you can test web applications simulta-
neously running in different browsers in a virtual lab. Ìn this case the
browsers are completely isolated from each other and this testing ap-
proach gives the most reliable results.
The following image demonstrates two instances of a web application
running in two browsers ÷ Ìnternet Explorer and Firefox. Ìnternet
Explorer is launched on a virtual machine running Windows Vista,
Firefox ÷ on a real machine running Windows 7. So, we can perform
application testing on machines with different browsers and confgura-
tions simultaneously and independently of each other.
Figure 6. Testing Web Applications on Physical and Virtual Machines
Testing Web Applications in a Virtual Lab with TestComplete
TestComplete
by SMARTBEAR
19
There are some obvious benefts of testing applications under virtual
platforms; however, they have some limitations too. The virtual operat-
ing system can work slower than a real computer, but this depends on
the situation: the virtualization technology, hardware, the number of
simultaneously running virtual platforms and others. Besides the speed
limitations, some hardware functionality can only be used by native
operating systems directly working on a physical computer. So, you
should take into account these virtual platform limitations, especially if
you are testing with specifc hardware features, multiprocessor systems
and other important factors. Of course, these limitations depend on the
particular VM software and hardware and you should check their docu-
mentation frst before performing automated testing.
There is a lot of VM software available on the market. TestComplete can
work with any virtual machine software, including the most popular for
which Automated Build Studio includes a special support:
Microsoft Virtual PC 2004 and 2007 (Microsoft Virtual PC
operations category)
Microsoft Virtual Server 2005 (Microsoft Virtual Server operations
category)
VMware Server 1.0 or later (VMware Server operations category)
VMware Workstation 5.5 or later (VMware Workstation operations
category)
No matter which VM software you use, you can easily automate your
QA testing in a virtual lab with TestComplete.
Virtual Machine Limitations
TestComplete
by SMARTBEAR
20
Automated testing in virtual labs and working with virtual machines al-
lows QA testers to get the maximum beneft out of their testing with the
lowest overhead cost. TestComplete's ability to work with both virtual
machines and real computers gives testers an edge and can even
increase your testing coverage and effciency. TestComplete also helps
you achieve your maximum ROÌ when using automated testing in virtual
labs. Ìf you haven't tried TestComplete, be sure to download and try it
today.
Ìf you are interested in automating the entire process of running the
tests on virtual machines, be sure also to install Automated Build Studio
on your local machine.
Ìf you want to learn how to use cloud computers in your testing, how to
prepare them and automate the testing process, see the Cloud Comput-
ers With TestComplete and Cloud Test Automation Tutorial articles on
our web site.
Conclusion
About SmartBear Software
More than one million developers, testers and operations profession-
als use SmartBear tools to ensure the quality and performance of their
APÌs, desktop, mobile, Web and cloud-based applications. SmartBear
products are easy to use and deploy, are affordable and available for
trial at the website. Learn more about SmartBear, the company's award-
winning tools or join the active user community at http://www.smartbear.
com, on Facebook or follow us on Twitter @smartbear and Google+.
SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915
+1 978.236.7900 www.smartbear.com
©2012 by SmartBear Software, Ìnc. Specifcations subject to change. SB-TC-020813-WEB
Start Creating Automated Functional
Tests for Web, Windows and Cloud Apps
Try TestComplete
Free 30-Day Trial