Professional Documents
Culture Documents
Chang Liu
Information and Computer Science
University of California, Irvine, Irvine, CA 92697, USA
liu@ics.uci.edu
713
release. For test automation engineers, these changes are the Apache web server1 on the host-under-test, and then it
equivalent to requirement changes if their task is viewed as tries to get a page from this host machine. After it gets the
software development. We all know, as a basic software page, it checks if a text string "It Worked!" is present in the
engineering problem, that requirement changes are hard to returned page. Various test execution tools may be used to
deal with when implementation is already underway. Not execute this test case automatically.
surprisingly, one of the biggest challenges of today's test <TestCase>
automation engineers is to keep pace with the ever- <Name>SimpleText</Name>
changing face of the application-under-test. <Action-RestartServer/>
<Action-GetPage>
Problem 3: Difficult to port <Url>$TestHost</Url>
Implementation changes are not the only source of <Oracle-Text>
<Text>It Worked!</Text>
"requirement changes" for test automation engineers. If the </Oracle-Text>
application-under-test is ported to another platform, the </Action-GetPage>
project team should perform the same automated testing on </TestCase>
the new platform. This presents test automation engineers a Fig. 1. A TestTalk test case from the Web Testing
significant challenge because: first, very few test Framework.
automation tools are cross-platform; second, even fewer TestTalk test descriptions describe only what software tests
test automation tools behave identically on different do and what they expect as desired outputs, not how to
platforms; third, the application-under-test is probably also execute the tests or how to verify the results. We
adjusted according to the different flavor of the new intentionally separate test automation details from test
platform. descriptions, because test automation details usually have
Switching test automation tools can also potentially force close ties to specific platforms or specific test execution
rewriting of all automated test cases. Switching is needed, tools. This separation ensures that TestTalk test
among other situations, when a new test automation tool descriptions are not sensitive to changes in application
comes up with a desirable feature that a team's old favorite implementation details and are portable.
tool lacks. While maintaining high understandability, maintainability,
3 OUR SOLUTION: TESTTALK and portability, TestTalk test descriptions are also
To alleviate the problems described in Section 2, we automation-friendly. For each test action and test oracle,
propose a platform-independent and tool-neutral testing testers can define how it can be implemented on various
language to describe automation-friendly software tests. platforms using various tools. One way to define this is to
The language that we developed for this purpose is write transformation rules that can translate TestTalk test
TestTalk [2,3], a comprehensive testing language based on actions and test oracles into complete or partial programs or
XML (http://belmont-shores.ics.uci.edu/TestTalk). scripts. For each testing tool on each platform, a set of
transformation rules are developed to implement every test
In TestTalk, a test case consists of a series of test actions action and oracle. With a transformation rule set, a test case
and test oracles. A test action represents a particular test description can be transformed to a program or script. This
step or a particular series of test steps that testers would generated program or script implement the test case on a
perform to carry out a test or a part of a test. A test oracle particular platform using a particular tool. To use another
represents a verification step or a series of verification steps testing tool or to test on another platform, only a new set of
that testers would perform to verify parts or all of test transformation rules is needed. All test descriptions remain
outputs. A tester can define customized test actions and test the same. Therefore, not only are TestTalk test descriptions
oracles suitable in her application domain. A collection of automation-friendly, the exact same test descriptions can
tester-defined tests actions and test oracles is called a also be automatically executed on different platforms using
dialect. A tester can use both her own dialect as well as different tools.
vocabulary provided by the TestTalk system to describe her
test suites and test cases. Since dialects are defined by 4 CASE STUDIES
testers in the field, who know the application domain better To validate the TestTalk language, we used it in a web
than language designers, the test descriptions are likely testing framework project. Currently, we are using
easier to write and easier to understand. TestTalk in two other case studies. In this section, we
briefly introduce these case studies.
An example TestTalk test case SimpleText is shown in Fig.
1. In this test case, there are only two simple test actions: Case Study 1: Web Testing Framework
RestartServer and GetPage. Only one simple test oracle The Web Testing Framework is a framework for Apache-
Text is used. When this test case is executed, it first restarts based web testing. The Web Testing Framework can be
1
Apache is an open-source HTTP server software package.
714
used by web developers to write test cases that can be essentially a set of protocols, implementations other than
automatically executed to test a web site. It can also be the HP implementation are expected. It is critical to quickly
used by developers of the Apache web server to quickly and accurately verify conformance of a new E-Speak
verify if a new build of the Apache server functions as implementation.
expected. Test cases in the Web Testing Framework are all
We are working on the E-Fluency Test project
written in TestTalk. Web developers or Apache developers
(http://belmont-shores.ics.uci.edu/E-Fluency), which is a
can use a set of predefined test actions and test oracles to
certification test suite for E-Speak. Because of the
quickly put together a test case. They can choose to execute
heterogeneous nature of E-Speak, E-Fluency Test has to be
the test case by lynx2, Expect3, lwp4, wget5, or curl6. They
platform independent. TestTalk is an ideal language to use
can also choose to execute the test case by any other tool
in such a situation.
that they think is suitable. Furthermore, they can even add
they own test actions and test oracles. To implement a new 5 RELATED WORK
test action or oracle, or to implement an existing test action Ostrand and Balcer's TSL (Test Specification language) is a
or oracle using a new tool, a web tester only needs to language designed to describe functional test suites by
provide a segment of Perl code that complies with a pre- specifying categories. It provides property and selector
defined interface protocol. This ability to incorporate new mechanism, which testers can use as constraints to annotate
code is also the reason why the project is named "Web the test specification so that they can control the
Testing Framework" instead of "Web Testing Tool". complexity and number of the tests. TSL does not address
Expandability was a major design goal. test oracles and test result verification.
Our experience with this project showed that TestTalk TDD (Test Data Definition language) from Sun
supported multi-platform and multi-tool test automation. Microsystems has three types of elements to help generate
more effective test data by specifying test categories. These
Case Study 2: Securibot
three types of elements are properties, test variables, and
Many computer systems were compromised after attackers
test directives. TDD does not provide support for test
exploited known security problems. To solve this problem,
automation.
we are developing a framework for automated security
checking and patching. The framework, named as 4Test is a commercial testing language from Segue. While
Securibot, provides a self-operating mechanism for security it supports a higher abstraction level so that test
checking and patching. Securibot performs security testing descriptions are not closely tied to the GUI of the
and analysis using security profiles and security updates, application-under-test, it is tool-specific.
both of which are specified in TestTalk. Securibot can also
detect compromised systems using attack signatures, also 6 SUMMARY
specified in TestTalk. Most important, the Securibot Software test automation has great potential in improving
framework allows system vendors to publish recently software quality and reducing software development costs.
discovered security weaknesses and new patches in However, understandability, portability, and
TestTalk so that the Securibot system running on deployed maintainability of executable test descriptions must be
systems can automatically check out security updates and addressed before the potential of test automation can be
apply the patches. fully realized. TestTalk is an attempt in this direction. The
Web Testing Framework case study shows that a
Case Study 3: E-Fluency Test for E-Speak customizable test framework can be built on top of
E-Speak is an on-going project at Hewlett-Packard TestTalk. We are working on two more case studies to
(http://www.e-speak.hp.com/). The goal is to provide collect further evidences.
uniform services over the Internet. E-Speak defines a
uniform service interface (API) and uniform service REFERENCES
[1] Chang Liu, Debra J. Richardson, "Programming
interaction (E-Speak engine) that allows e-services to
Languages Considered Harmful in Writing Automated
dynamically interact to discover, negotiate, broker and
Software Tests", Technical Report 99-09, Inform. &
compose themselves to solve a business to business or
Comp. Sci., University of California, Irvine, Feb. 1999.
business to consumer service request. Since E-Speak is
[2] Chang Liu, Debra J. Richardson, "TestTalk, A Test
2
Description Language: Write Once, Test by Anyone,
Lynx is a text-based web browser. Anytime, Anywhere, with Anything", Technical Report
3
Expect is a testing tool based on scripting language TCL. 99-08, Inform. & Comp. Sci., UC Irvine, Feb. 1999.
4
Lwp is libwww-perl, the Perl World-Wide-Web library
(http://www.perl.com/CPAN). [3] Chang Liu, "TestTalk: A Comprehensive Testing
5
GNU wget is a network utility to retrieve files from the Language", the 14th IEEE International Conference on
World Wide Web. Automated Software Engineering, Doctoral
6
Curl is a command line tool for getting data from a URL. Symposium, Oct. 1999, Cocoa Beach, Florida, USA.
715