Distributed Software Testing

MAIMUUNMUNTANT DE LA UNITA ATMATI HAITI
US009811451B1
(12) United States Patent (10) Patent No.: US 9 ,811,451 B1

Arguelles et al. (45 ) Date of Patent: Nov. 7 , 2017
(54) DISTRIBUTED SOFTWARE TESTING (56 ) References Cited
(71 ) Applicant: Amazon Technologies, Inc ., Seattle, U .S . PATENT DOCUMENTS
WA (US ) 8,769,102 B1* 7/2014 Zhou .................. GO6F 11/3696
709 /223
(72) Inventors : Carlos Alejandro Arguelles, Shoreline , 2006 /0025953 A1 * 2 /2006 Janes ............... GO1R 31 /31707
702 / 108
WA (US); David Robert Griffin ,
Seattle , WA (US) * cited by examiner
( 73 ) Assignee : Amazon Technologies, Inc ., Seattle , Primary Examiner — Chuck Kendall
WA (US )
(74 ) Attorney , Agent, or Firm — Lee & Hayes , PLLC
(57) ABSTRACT
( * ) Notice : Subject to any disclaimer, the term of this Technologies are disclosed herein for distributed software
patent is extended or adjusted under 35 testing . A software testing service is disclosed that can
U .S .C . 154 (b ) by 0 days. receive tests for software under test . The software testing
service can determine a number of test instances to utilize to
(21) Appl. No.: 14 /857,252 perform the tests . The determined number of test instances
can then be provisioned , such as through the use of an
(22 ) Filed : Sep . 17, 2015 on -demand computing service executing in a service pro
vider network . The test instances can also be configured with
the software under test and a test runner capable of per
(51) Int . Cl. forming the tests on the software. The software testing
G06F 9 /45 ( 2006 .01) service can place the tests on a queue, such as a queue
G06F 11/ 36 ( 2006 .01) provided by a queue service . The test runners executing on
U .S . CI. the test instances can dequeue the tests and perform the tests
CPC ... G06F 11/3688 (2013.01) on the software . Once the testing of the software under test
(58 ) Field of Classification Search has been completed , the test instances can be de- provi
None sioned
See application file for complete search history. 20 Claims, 11 Drawing Sheets
TEST RUNNER
- 400
START
POLL QUEUE FOR A TEST 402
NO
WAIT TEST ON QUEUE ?
YES
406 DEQUEUE TEST FROM 408
THE TEST QUEUE
PERFORM THE TEST ON THE 410

SOFTWARE UNDER TEST
GENERATE AND STORE TEST 412

RESULTS , INCLUDING TIME TO
PERFORM THE TEST
U . S . Patent Nov . 7, 2017 Sheet 1 of 11 US 9,811, 451 B1
126 1 2A 24A 112B 724B 1240126 1120 124D

sedurres
T1 6
H
H
RTUENSTR TEST TEST USONFTDWEAR ITNSEANCTE RTUENSTR SUONFTDWEAR TEST ITNSEANCTE ITNSEACT RTUENSETR TEST SUONFTDWEAR TEST ITNSEACT SCOEMRPVUTICNEG
ON-DEMAND
.
.
r
- - - -
120
RT12E8S?TFESULT TESTRESULT TESTRESULT RTESULTS STORAGE 1 2T-AETSESTT 11284TTESTTEST TEST TEST QUE QUE SERVICE
)
TEST
-
1
.
FIG
128B 128C
1120 1 82
104
SOFTWAR TESINGSERVICE HISTORCAL EXCUTION NPSREOTVWIDRCEK
TEST TDIAMTEA
126
TEST TESING SPREF NCS OFTWARETUNEDSTR RTEQUSET
102
106
DEVL RCOMPUTINGDEVICE SOFTWAREDVLOPER

atent Nov . 7 , 2017 Sheet 2 of 11 US 9,811 ,451 B1
SOFTWARE TESTING SERVICE

200
START
RECEIVE TEST REQUEST 7 202
202
TEST TM 204
DETERMINE NUMBER OF TEST
INSTANCES TO USE FOR TESTING 204
(FIGURE 3 )
UTILIZE ON -DEMAND COMPUTING 206

SERVICE TO PROVISION THE TEST
INSTANCES
DEPLOY TEST RUNNER 208

AND SOFTWARE UNDER TEST
PLACE THE TESTS ON THE TEST QUEUE

RECEIVE TEST RESULTS FROM THE 212
TEST RUNNERS INDICATING THE TIME IT
TOOK THE TESTS TO COMPLETE
UPDATE THE HISTORICAL TEST 214

EXECUTION TIME DATA
216
NO
ALL TESTS COMPLETED ?
YESI
DE -PROVISION THE TEST INSTANCES 218
END y 220
FIG . 2
U . S . Patent Nov . 7, 2017 Sheet 3 of 11 US 9 ,811 ,451 B1
DETERMINE NUMBER OF TEST INSTANCES

TO USE FOR TESTING
(OPERATION 204 , FIG . 2 ) - 300
START
USE THE 302
SPECIFIED | YES NUMBER
NUMBER OF TEST OF TEST INSTANCES 308
SPECIFIED ?
INSTANCES
NOL
306
MAX EXECUTION « NO USE A DEFAULT
312 NUMBER OF TEST
TIME SPECIFIED ? INSTANCES
USE HISTORICAL
TEST EXECUTION
TIME DATA FOR
YEST 1 310
THE TESTS TO HISTORICAL *
COMPUTE AN TEST EXECUTION TIME DATA
ESTIMATE OF THE AVAILABLE FOR THE
TESTS ?
TIME TO
COMPLETE THE NO
TESTS 316
HISTORICAL X
USE THE ESTIMATE TEST EXECUTION TIME DATA
OF COMPLETION AVAILABLE FOR SIMILAR
TIME TO COMPUTE TESTS ?
THE NUMBER OF
TEST INSTANCES YES
NEEDED TO USE HISTORICAL TEST EXECUTION
PERFORM THE TIME DATA FOR SIMILAR TESTS TO
TESTS WITHIN THE
SPECIFIED MAX COMPUTE AN ESTIMATE OF THE
EXECUTION TIME , TIME TO COMPLETE THE TESTS
314
RETURN
Ca.2)
( TO OPERATION 206 , FIG . 2 )
FIG . 3 320
TEST RUNNER
START
402
7400
- 400
POLL QUEUE FOR A TEST

NO
WAIT UEUE? x
TEST ON QUEUE ?
YES
DEQUEUE TEST FROM 408

THE TEST QUEUE
PERFORM THE TEST ON THE

SOFTWARE UNDER TEST
GENERATE AND STORE TEST 4 412

RESULTS , INCLUDING TIME TO
PERFORM THE TEST
FIG . 4
U . S . Patent Nov. 7, 2017 Sheet 5 of 11 US 9,811, 451 B1
DYNAMICALLY ADJUST NUMBER

OF TEST INSTANCES DURING TEST EXECUTION
START -500
502
OBTAIN DATA IDENTIFYING TESTS THAT
HAVE BEEN COMPLETED , TESTS THAT
ARE CURRENTLY EXECUTING , TESTS
NOT YET EXECUTED , ELAPSED TIME ,
AND DESIRED DURATION OF TESTING
COMPUTE AN ESTIMATE OF THE 504

AMOUNT OF TIME NEEDED TO
COMPLETE THE TESTS
506
DE -PROVISION ONE YES WILL TESTS
OR MORE OF THE COMPLETE SOONER THAN
TEST INSTANCES DESIRED ?
NO
510
WILL TESTS NO
508 COMPLETE LATER THAN
DESIRED ?
YES
512
PROVISION AND CONFIGURE ONE OR
MORE ADDITIONAL TEST INSTANCES
FIG . 5
112A1 2A 126 1 2B 16B 724B

112B 126 1120 1 6C 7240
+
SUHTO1FNTD6WAERA TESTIN24ASTANCE
H
TRAT126UENSETR
TESTTEST TEST RTUENSETR TEST TEST USONFTDWEAR ITNESATCE RTUENSTR USONFTDWEAR ITNSEANCTE INSTACE
TEST TEST OFTFSLEATSIC
602
120
TEST TEST TEST TEST QUE QUE SERVICE 6
.
FIG
1 2A 1 2B 1 2C
104
104
SOFTWARETESTINGSERVICE
116B
1 2A
1 4A
71 61 1 2B 1 4B
1 20
1 40 116C
1 0A TEST TES ING PREF NCS SOFTWAREUTNEDSRT RTEQUST TEST TESING SPREF NCS OFTWARETUNEDSTR RTEQUST 1 0C TEST TESING SPREF NCS OFTWARETUNEDSTR RTEQUST
110B
(
108A DEVLOPERCMPUTINGDEVICE 108B DEVLOPRCMUTINGDEVICE 1080 DEVLOPRCMUTINGDEVICE

U . S . Patent Nov. 7, 2017 Sheet 7 of 11 US 9,811, 451 B1
DISTRIBUTION OF TESTS FOR MULTIPLE CLIENTS

TO A SHARED STATIC FLEET OF TEST INSTANCES
700
START
RECEIVE TESTS FROM MULTIPLE 702
CLIENTS OF THE SOFTWARE TESTING
SERVICE
UTILIZE HEURISTICS TO SELECT A 704

SUBSET OF THE TESTS
PLACE THE SUBSET OF THE TESTS ONT 706

THE TEST QUEUE
YES
708
WAIT MORE TESTS ?
NO
710
END
END
2 712
FIG . 7
U . S . Patent Nov. 7, 2017 Sheet 8 of 11 US 9,811,451 B1
804A 804B 8040 804D

4
- -- - - -- -- -- -
4804B - - - - - - - - -- -- -- - - - - - - - - - - - - - - - -
CDEANTEAR PSREOVIDCERNTWORK
-
CDEANTEAR CDEANTEAR CDEANTEAR -
-
-
- - - - - - - - - - - - - - - - - - - - - - - -
NETWORK 802
8
.
FIG
COMPUTINGDEVICE 800
atent Nov . 7 , 2017 Sheet 9 of 11 US 9,811, 451 B1
9060 906E
SERV COMPUTER NETWORKSERVICE SERVERCOMPUTER NETWORK SERVICE

STOEFRVWIACNEG CSOEMRPUVT
RESOURCE RESOUC
904D 904E
902D 9025 104 902F
web
lgorite
a
aan ALORCEAL NETWORK
ET NETWORK 9
.
FIG
CDEANTEAR 804 908
TOWIDEAREA
902A 902B 9020
906A 906C
1906B
SERV COMPUTER NETWORKSERVICE SERV COMPUTER NETWORK SERVICE SERV COMPUTER NETWORKSERVICE
padhe
monet
rtnering
RESOUCE RESOUCE RESOUCE
904A 9048 9040
9061
OOOOOOOOOO
906J OOOOOOOOOO :
104
- - - -
120
- - - -
.
V .
AUTHENICO MANGET
. .
. .
.
.
POLICY SERVICE SOFTWAR STERVICNEG SQEURVICE

. .
SERVICE
.
NPSREOTVWIDCKER
.
.
10
.
FIG
.
DEPLOYMNT CRYPTOGAH
.
SCOMPUTING ERVICE STORAGE SRVICE SERVICE SERVICE

.
ON-DEMAND
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
122 906F 906H 1024

906G
atent Nov . 7 , 2017 Sheet 11 of 11 US 9,811, 451 B1
908
NETWORK
1116
1112
NETWORK INPUT/
INTERFACE OUTPUT
1102 CONTROLLER CONTROLLER
1106
1106
HE
CHIPSET
1114
STORAGE
CPU ( S ) RAM ROM
CONTROLLER
1104 1108 1110

1100
1120
OPERATING SYSTEM
1122
PROGRAMS
FIG . 11
US 9 ,811,451 B1
DISTRIBUTED SOFTWARE TESTING for multiple clients of the software testing service are
distributed to a shared static fleet of test instances ;
BACKGROUND FIG . 7 is a flow diagram showing a routine that illustrates
aspects of the operation of the software testing service for
Thorough testing is one of the most important aspects of 5 distribution of tests for multiple clients to a shared static fleet
software development. In order to test complex software of test instances;
thoroughly , it is not unusual for a software developer to run FIG . 8 is a system and network diagram that shows an
thousands of tests, someof which might take hours or more illustrative operating environment that includes a service
to execute . If a software development team utilizes such provider network that can be configured to implement
tests to approve changes to the software , the extensive aspects of the functionality described herein ;
amount of time required to perform the tests might limit the FIG . 9 is a computing system diagram illustrating a
team ' s ability to make a code change and have the change configuration for a data center that can be utilized to
be deployed to a production environment in a timely manner. implement aspects of the technologies disclosed herein ;
This can be frustrating for the software developers on the 1 FIG . 10 is a system and network diagram that shows
software development team . aspects of several services that might be provided by and
As a result of the sometimes lengthy time required to utilized within a service provider network in one configu
perform software tests, some software development teams ration disclosed herein ; and
end up reducing the number of tests that are performed on FIG . 11 is a computer architecture diagram showing an
the software. This means that code changes reach a produc - 20 illustrative computer hardware architecture for implement
tion environment quicker. However, this also means that the ing a computing device thatmight be utilized to implement
risk that the software includes bugs is greater than if more aspects of the various technologies presented herein .
thorough testing had been performed . A higher risk of
deploying software containing bugs is unacceptable in many DETAILED DESCRIPTION
environments . 25
One solution to the problemsdescribed above is to utilize The following detailed description is directed to technolo
test platforms that allow for multithreaded execution of gies for distributed software testing . Through an implemen
tests . Using such a platform , multiple tests can be executed tation of these technologies, developer -created tests for
concurrently in the same process. However, this solution software can be distributed to a multitude of test instances
often has its own limitations. For example , in order to 30 for execution in parallel. By executing software tests in
perform multithreaded execution of tests , the test developer parallel in this manner, the amount of time required to
must write the tests in a way that ensures that the tests are execute the tests can be reduced as compared to non
safe for multithreaded execution . As a result, creating tests parallelized execution . Moreover, by executing software
for multithreaded execution might require significant addi- tests in parallel, the extensive testing needed to ensure high
tional work by the software developer. Additionally , multi- 35 quality software can be performed while , at the same time,
threaded execution only scales to the limits of the computing allowing the software to be deployed to a production envi
resources ( e. g . CPU and memory ) of the computer upon ronment in a timely manner. Additionally , the technologies
which they are executed . As a result , the benefits of multi disclosed herein enable the parallel execution of tests for
threaded execution of tests might be limited . software without requiring that the software developer
The disclosure made herein is presented with respect to 40 modify their existing tests in any way. Additional technical
these and other considerations . benefits other than those described briefly above can also be
realized through an implementation of the technologies
BRIEF DESCRIPTION OF THE DRAWINGS disclosed herein .
In order to enable the functionality disclosed herein , a
FIG . 1 is a system architecture diagram showing aspects 45 software testing service is provided in one configuration .
of the configuration and operation of a service provider The software testing service is a network service that can
network that is configured to provide a software testing execute in a service provider network that permits customers
service , according to one configuration disclosed herein ; to purchase and utilize computing resources (which might be
FIG . 2 is a flow diagram showing a routine that illustrates referred to herein simply as " resources ” ) such as virtual
aspects of the operation of the software testing service 50 machine (“ VM " ) instances , data storage resources, database
shown in FIG . 1, according to one particular configuration resources, networking resources, network services, and
disclosed herein ; other types of computing resources on demand . The com
FIG . 3 is a flow diagram showing a routine that illustrates puting resources are provided by network services executing
aspects of the operation of the software testing service in the service provider network . For example , and without
shown in FIG . 1 for determining a number of test instances 55 limitation , an on - demand computing service executing in a
to utilize for testing , according to one particular configura service provider network can be configured to provide
tion disclosed herein ; computing resources such as VM instances on an as -needed
FIG . 4 is a flow diagram showing a routine that illustrates basis . Other services can also provide other types of com
aspects of the operation of the test runner illustrated in FIG . puting resources on an as -needed basis .
1 , according to one particular configuration disclosed herein ; 60 The software testing service disclosed herein is config
FIG . 5 is a flow diagram showing a routine that illustrates ured to receive a request to perform tests on software (which
aspects of the operation of the software testing service might be referred to herein as " software under test” ). The
shown in FIG . 1 for dynamically adjusting the number of request can include the software or a reference to the
test instances used for testing during test execution , accord software , the tests or references to the tests and , potentially ,
ing to one particular configuration disclosed herein ; 65 one or more testing preferences. The request might, for
FIG . 6 is a system architecture diagram showing another example , be generated and submitted to the software testing
configuration of the software testing service in which tests service by a software developer of the software under test.
US 9,811 ,451 B1
The developer-specified testing preferences can define desired amount of time for completion (i.e . the tests will
various preferences regarding how the tests are to be dis complete later than desired), then additional test instances
tributed to test instances for execution in parallel. For can be provisioned and configured in the manner described
example , and without limitation , the testing preferences above. In this way, the number of test instances can be
might explicitly specify the number of test instances to 5 dynamically optimized during testing in order to complete a
utilize for performing the tests on the software . As another suite of tests approximately within a specified amount of
example , the testing preferences might specify a maximum time.
execution time for the tests . In this scenario , the number of In another configuration , the software testing service is
test instances to utilize to complete testing approximately configured to operate in conjunction with a static fleet of test
within the maximum execution time can be determined 10 ;instances. In this configuration , the number of available test
based upon historical test execution time data for the tests. instances is fixed and cannot be modified dynamically as in
Alternately , if historical test execution time data is not the configuration described above . Additionally , in this
available for the tests , the number of test instances to utilize configuration the software testing service can be configured
to complete testing within approximately the maximum
execution time can be determined based upon historical test 15 toto distribute tests for multiple different service clients (e .g .
execution time data for other tests thatnces
aremight
similaralso
to the customers) to the fixed number of test instances. For
tests
specify
to be performed . The testing preferences might also specify example , and without limitation , the software testing service
other or alternate preferences regarding the manner in which might receive requests from multiple service clients to
the tests are to be performed . perform tests on different software .
Once the number of test instances to utilize for testing has 20 Utilizing one or more heuristics , the software testing
been determined , the software testing service can cause the service can select a subset of the tests for one of the service
test instances to be provisioned . For example , in one con - clients and a subset of the tests for another of the service
figuration , the software testing service can utilize an on - clients . The selected subsets of the tests can then be made
demand computing service executing in the service provider available to a test runner executing on the test instances in
network to provision the test instances. The test instances 25 the static fleet. Once the tests have been completed , the
can be VM instances , software containers , dedicated server heuristics might be utilized to select another subset of the
computers , or other types of software execution environ - tests, which are then made available to the test runners and
ments suitable for executing tests on software under test performed on the software under test. This process can
Once the test instances have been provisioned , the software continue until no further tests remain to be performed .
under test and a test runner can be deployed to the test 30 The heuristics utilized to select subsets of the tests for
instances . The test runner is executable software configured performance on the software under test can include , but are
to obtain tests , perform the tests on the software under test, not limited to , a priority associated with the tests , a desired
and to provide the results of the tests . completion time for the tests, and an estimated time for
In one particular configuration, the software testing ser completion of the tests. Other heuristics might also or
vice is configured to utilize a queue provided by a queue 35 alternately be utilized in other configurations. Additional
service executing in the service provider network to provide details regarding the various components and processes
the tests to the test runner executing on the test instances . For described briefly above for distributed software testing will
example , the software testing service may place the tests , or be presented below with regard to FIGS. 1 - 11 .
messages including references to the tests , on such a queue . It should be appreciated that the subjectmatter presented
The test runner executing on the test instances can dequeue 40 herein can be implemented as a computer process , a com
the tests from the queue and perform the tests on the puter - controlled apparatus , a computing system , or an article
software under test. The test runner will continue dequeue of manufacture , such as a computer - readable storage
ing tests and executing the tests until no further tests remain medium . While the subject matter described herein is pre
on the queue. sented in the general context of software modules that
The software testing service can also determine whether 45 execute on one or more computing devices, those skilled in
all of the tests have completed . If all of the tests of the the art will recognize that other implementations can be
software under test have completed , the test instances can performed in combination with other types of software
then be de -provisioned . For example , and without limitation , modules . Generally , software modules include routines,
the on -demand computing service executing in the service programs, components , data structures, and other types of
provider network mightbe instructed to de-provision the test 50 structures that perform particular tasks or implement par
instances . In this way , the test instances are only utilized ticular abstract data types .
during testing and can be destroyed once testing has com Those skilled in the art will also appreciate that aspects of
pleted . the subject matter described herein can be practiced on or in
In some configurations, the number of test instances conjunction with other computer system configurations
utilized for testing can be dynamically increased or 55 beyond those described herein , including multiprocessor
decreased during testing . For example, and without limita systems, microprocessor -based or programmable consumer
tion , the software testing service can be configured to electronics , minicomputers , mainframe computers, hand
compute an estimated amount of time for completion of the held computers , personal digital assistants , e -readers ,mobile
tests . The software testing service can also determine telephone devices, tablet computing devices, special- pur
whether the estimated amount of time for completion of the 60 posed hardware devices , network appliances, and the like.
tests is greater than or less than a desired ( e. g . developer - As mentioned briefly above , the configurations described
specified ) amount of time for completing the tests . herein can be practiced in distributed computing environ
If the estimated time for completion of the tests is less ments, such as a service provider network , where tasks can
than the desired amount of time for completion ( i. e. the tests be performed by remote computing devices that are linked
will complete sooner than desired ), then one or more of the 65 through a communications network . In a distributed com
test instances can be de - provisioned . If, however, the esti - puting environment, program modules can be located in
mated time for completion of the tests is greater than the both local and remote memory storage devices.
US 9,811 ,451 B1
In the following detailed description , references are made (“ WANs” ), such as the Internet. Additional details regarding
to the accompanying drawings that form a part hereof, and the configuration and operation of a data center for imple
that show , by way of illustration , specific configurations or menting the functionality disclosed herein are provided
examples . The drawings herein are not drawn to scale . Like below with regard to FIG . 9 .
numerals represent like elements throughout the several 5 The resources described briefly above might also be
figures (which might be referred to herein as a “ FIG .” or provisioned and de - provisioned as needed in an automated
“ FIGS.” ) . fashion . For example , the service provider network 102
FIG . 1 is a system architecture diagram showing aspects might be configured to create a new instance of a computing
of the configuration and operation of a service provider resource , such as a VM instance, in response to an increase
network 102 that is configured to provide functionality for 10 in demand for a network service or other condition . Other
distributed software testing. As discussed briefly above, the types of computing resources might also be provisioned and
service provider network 102 is a distributed network de - provisioned in a similar manner. Services in the service
through which customers and / or other users can utilize provider network 102 might also provide functionality for
computing resources (which might be referred to herein automatically scaling and / or de - scaling resources based
simply as " resources ” ) , such as VM instances and/ or other 15 upon demand for the resources and /or other factors .
types of computing resources , on a permanent or as - needed A customer or potential customer of the service provider
basis . network 102 might utilize an appropriate computing system ,
Each type or configuration of a computing resource can be such as the developer computing device 108 , to communi
available from the service provider network 102 in different cate with the service provider network 102 over an appro
sizes . For example , a service provider might offer physical 20 priate data communications network (not shown in FIG . 1 ).
hosts , VM instances or other types of data processing In this way , a customer of the service provider network 102
resources that are available for purchase and use that have can configure various aspects of the operation of the com
many different configurations ofprocessor capabilities , main puting resources provided by the service provider network
memory, disk storage , and operating system . A service 102 , or to otherwise control any computing resources being
provider operating the service provider network 102 might 25 utilized by the customer.
also offer other types of computing resources for purchase For example , and without limitation , a computing system
and use by customers . For example , a service provider might utilized by a customer of the service provider network 102,
offer virtual or hardware devices , database resources and such as the developer computing device 108 , might be
instances, file or block data storage resources, and / or net - utilized to purchase computing resources in the service
working resources , such as load balancing resources, 30 provider network 102 , to configure aspects of the operation
domain name service (“DNS” ) resources, virtual private of the computing resources through a management console
cloud (“ VPC ” ) resources , virtual local area network (not shown in FIG . 1) or other type of interface, to access
(“ VLAN " ) resources, and / or other types of hardware and and utilize functionality provided by the various services
software computing resources or services on a permanent or and systems described herein , and /or to perform other types
as-needed basis . The computing resources might also 35 of functionality with regard to the operation of the comput
include , but are not limited to , VM instances and images, ing resources provided by the service provider network 102 .
security groups , option groups, gateways , option sets, net . The developer computing device 108 mightbe any type of
work access control lists (" ACLs" ) , subnets , storage buck - computing device capable of connecting to the service
ets, network interfaces, queues , snapshots, spot market provider network 102 via a suitable data communications
requests , and storage volumes . 40 network such as , but not limited to , a laptop or desktop
The service provider operating the service provider net - computer, a tablet computing device , a server computer, or
work 102 might also charge a fee for utilization of the a smartphone. Administrative users employed by the opera
computing resources to a customer that creates and uses the tor of the service provider network 102 , such as adminis
resources . The fee charged for a particular computing trators managing the operation of the service provider net
resource might be based upon the type and / or configuration 45 work 102 , might also connect with , manage , and utilize
of the computing resource . For example , in the case of a data resources provided by the service provider network 102 in a
processing computing resource, like a VM instance , the fee similar fashion .
for use of the computing resource might be charged based As discussed briefly above, the service provider network
upon the amount of time the computing resource is utilized . 102 might also be configured to execute various types of
In the case of a data storage computing resource , the fee 50 network services to provide the computing resources 108 .
might be computed based upon the amount of data stored For example , and without limitation , the service provider
and /or the amount of data transferred into or out of the network 102 can provide an on -demand computing service
computing resource . The fees for other types of resources 122 (described in greater detail below ) for providing VM
might also be based upon other considerations. A service instances or other types of processing resources on - demand,
provider might also utilize various purchasing models to 55 a queue service 120 , a data storage service (not shown in
determine the amount to charge a customer for use of FIG . 1 ) for providing data storage resources and , potentially ,
resources provided by the service provider network 102 . other types of network -accessible services for providing
The resources described above can be provided in one other types of computing resources . These and other net
particular implementation by one or more data centers work services and their associated computing resources can
operated by the service provider . As known to those skilled 60 be utilized together to implement various types of network
in the art, data centers are facilities utilized to house and based applications in the service provider network 102 .
operate computer systems and associated components . Data As discussed briefly above, a software testing service 104
centers also typically include redundant and backup power, also executes in the service provider network 104 in one
communications, cooling , and security systems. The data configuration. The software testing service 104 is configured
centers might be located in geographically disparate regions, 65 to receive a request ( e . g . the test request 110 shown in FIG .
and might also be connected to various other facilities , such 1) to perform tests 112 on software 116 (which might be
as co - location facilities, and various wide area networks referred to herein as a " software under test” ). The test
US 9,811,451 B1
request 110 can include the software 116 or a reference to the the tests 112 . The historical test execution time data 126
software 116 to be tested , the tests 112 or references to the describes the average time taken to execute the tests 112
tests 112 and , potentially , one or more testing preferences during previous testing .
114 . The test request 110 might, for example, be generated It should be appreciated that the testing preferences 114
and submitted to the software testing service 104 by a 5 described above are merely illustrative . Accordingly, the
testing preferences 114 might also specify other or alternate
software developer 106 of the software 116 under test . Other preferences
services in the service provider network 102 can also submit to be performed regarding the manner in which the tests 112 are
test requests 110 to the software testing service 104 . . For example , and without limitation , the
The tests 112 are software components that are configured 10 testing preferences 114 might specify the type of test
instances 124 to be used , a maximum cost to be incurred for
to test various aspects of the operation of the software 116 executing the tests 112 , and / or other information .
under test. For example , and without limitation , the tests 112 In order to compute the number of test instances 124 to
can include unit tests that are configured to tests functional utilize based on the historical test execution time data 126
areas of the software 116 . Other types of tests 112 can also for the tests 112 , the software testing service 104 utilizes the
be defined . The tests 112 can be defined utilizing existing 15 historical test execution time for a test 112 as an estimate of
testing frameworks. For example, and without limitation, the how long the test 112 will take to run again . When the
tests 112 can be defined utilizing the JUNIT or TESTNG execution time for a test 112 changes , the software testing
testing frameworks . Other frameworks can also be utilized service 104 can readjust the estimated time for execution
By utilizing tests 112 that are compatible with existing utilizing a mechanism whereby weights are assigned to
testing frameworks, the software developer 106 is not 20 execution times. In particular, a window of N executions of
required to create new tests 112 that are compatible with the a test 112 might be utilized , and more recent executions of
software testing service 104 disclosed herein . the test 112 will be assigned a higher weight than older
The tests 112 can also be defined as encompassing mul executions . Other mechanismsmight also be utilized .
tiple distinct tests . In this way, multiple tests 112 that need If historical test execution time data is not available for
to be executed sequentially on the same test instance 124 25 the tests 112 , the number of test instances 124 to utilize to
(which might be referred to herein as the “ smallest execution complete executing of the tests 112 within approximately the
unit ) can be defined . Industry standard testing frameworks , specified maximum execution time can be determined based
such as JUNIT and TESTNG . provide functionality for upon historical test execution time data for other tests that
modeling dependencies between individual tests 112 . The 30 are similar to the tests 112 to be performed . For example ,
software testing service 104 understands and respects these 30 and without limitation , the other tests might be tests in the
dependencies . For example , and without limitation , the same class , package , or archive . In this example , historical
execution times for tests in the same class might be given a
software developer 106 can specify that a first test 112 greater weight than tests in the same package . Similarly ,
depends upon a second test 112 . This might be true when historical execution times for tests in the same package
first test 112 creates data that the second test 112 uses or 35 might be assigned a greater weight than the historical
when the second test 112 is only to be executed when the execution times of tests in the same archive. In this manner ,
first test 112 passes. In either case , the software testing the historical execution times for other tests can be assigned
service 104 understands that both the first test 112 and the a weight that is based upon their relevance to the tests 112
second test 112 cannot be executed independently or in to be performed . Other mechanisms can also be utilized to
parallel. Rather, both tests 112 need to be executed sequen - 40 estimate the time to complete the tests 112 from historical
tially on the same test instance 124 . In order to enable this execution times of other tests .
functionality , the software testing service 104 will group the Utilizing the historical test execution time data 126 for the
first test and the second test for execution on a single test tests 112 , or for other similar tests , the software testing
instance 124 . service 104 can compute an estimate of the amount of time
In this regard , it should be appreciated that various 45 it will take to execute the tests 112 sequentially and approxi
mechanisms can be utilized to model the smallest execution mately how long it will take to run the tests 112 in parallel
unit . In the case of the JAVA programming language , for o n N test instances 124 . Likewise , the software testing
example , a JAVA class or a JAVA package can be defined as service 104 can compute how many test instances 124 it
the smallest execution unit. Industry standard frameworks would take to execute the tests 112 in M minutes . For
such as TESTNG and JUNIT also support the concept of 50 example , and without limitation , if the tests 112 have
" suites ” , typically by adding JAVA annotations on a test 112 historically taken approximately three hours to execute , it
to indicate the suite to which it belongs. Other mechanisms would require three test instances 124 to perform the tests
can also be utilized to model the smallest execution unit in 112 in one hour. Likewise, if the tests 112 were executed on
other configurations. ten test instances 124 , it would take approximately eighteen
As mentioned briefly above , the test request 110 can also 55 minutes to execute the tests 112 . Additional details regarding
specify testing preferences 114 . The testing preferences 114 one mechanism disclosed herein for determining the number
can define various preferences regarding how the tests 112 of test instances 124 to utilize for testing will be provided
are to be distributed to test instances 124 for execution in below with regard to FIG . 3 .
parallel. For example , and without limitation , the testing Once the software testing service 104 has computed the
preferences 114 might explicitly specify the number of test 60 number of test instances 124 to utilize for testing, the
instances 124 to utilize for performing the tests 112 on the software testing service 104 can cause the test instances 124
software 116 . As another example , the testing preferences to be provisioned . For example , in one configuration , the
114 might specify a maximum execution time for the tests software testing service 104 can utilize an on -demand com
112 . In this scenario , the number of test instances 124 to puting service 122 executing in the service provider network
utilize to complete execution of the tests 112 approximately 65 102 to provision the identified number of test instances 124 .
within the specified maximum execution time can be deter - As discussed briefly above , the test instances 124 can be VM
mined based upon historical test execution time data 126 for instances , software containers , dedicated server computers ,
US 9,811 ,451 B1
10
or other types of software execution environments suitable device 108 to access and examine the test results 128 .
for executing tests on a software 116 under test . In the Additional details regarding the operation of the test runner
example shown in FIG . 1, the software testing service 104 126 will be provided below with regard to FIG . 4 .
has instructed the on -demand computing service 122 to The test results 128 might also be provided to the software
provision three test instances 124A - 124C . In this regard , it 5 testing service 104 . In some configurations, the test results
should be appreciated that fewer or more test instances 124 128 include various tops of logs and /or other types of data
might also be utilized than shown in FIG . 1. describing aspects of the execution of the tests 112 . The test
Once the test instances 124 to be utilized for testing the results 128 can also include data indicating the time of
software 116 have been provisioned , the software 116 under execution of the corresponding test 112 . Utilizing this data ,
test and a test runner 126 can be deployed to the provisioned 10 the software testing service 104 can update the historical test
test instances 124 . As discussed briefly above, the test runner execution time data 126 . For example , the software testing
126 is executable software configured to obtain the tests 112 , service 104 can store data identifying the test 112 that was
perform the tests 112 on the software 116 under test, and to executed , the amount of time taken to execute the test , the
provide the results of the tests 112 . Each of these operations class or other functional unit of tested software , and /or other
is described in greater detail below . In this regard , it should 15 data describing other aspects of the execution of the test 126 .
be appreciated that the test runner 126 might be configured As discussed above, this data can be utilize to estimate the
to execute tests 112 that have been created in accordance time to execute the tests 112 or other tests .
with a particular testing framework . For example , and The software testing service 104 can also determine
without limitation , the test runner 126 can be configured to whether all of the tests 112 have completed . For example ,
perform tests 112 that have been created in accordance with 20 and without limitation , the software testing service 104
the JUNIT or TESTNG testing frameworks . The test runner might examine the test queue 118 or the test results 128 to
126 might also be compatible with tests 112 that are com determine if all of the tests have completed . Alternately, the
patible with other testing frameworks in other configura - test runner 126 can provide other types of notifications to the
tions. software testing service 104 including, but not limited to ,
It should also be appreciated that the software under test 25 status messages and messages indicating that tests 112 have
can be present on the same test instance 124 as the test completed .
runner 126 or can be installed on a different instance 124 . If all of the requested tests 112 of the software 116 under
For instance , in the example shown in FIG . 1, the software test have completed , the test instances 124 utilized for
under test is present on the instances 124A - 124B . This testing can be de- provisioned. For example , and without
configuration might be appropriate where the tests 112 are 30 limitation , the software testing service 104 can instruct the
unit test , for example. The software under test has also been on -demand computing service 122 executing in the service
installed on a separate test instance 124D for testing by the provider network 102 to de - provision the test instances 124 .
test runner 126 executing on the test instance 126C . This In this way, the test instances 124 are only utilized during
configuration might be appropriate where the tests 112 are testing and can be destroyed once testing of the software 116
integration or performance tests . Other types of tests 112 and 35 under test has completed . Additional details regarding the
configurations can also be utilized . operation of the software testing service 104 will be pro
In one particular configuration , the software testing ser vided below with regard to FIG . 2 .
vice 104 is configured to utilize a distributed queue 118 In some configurations, the number of test instances 124
provided by a queue service 120 executing in the service utilized for testing can be dynamically increased or
provider network 102 to provide the tests 112 (or references 40 decreased during testing . For example , and without limita
to the tests 112) to the test runner 126 executing on the test tion , the software testing service 104 can be configured to
instances 124 . For example , the software testing service 104 compute an estimated amount of time for completion of the
may place the tests 112 , or messages including references to tests 112 . The estimated amount of time for completion of
the tests 122 , on such a queue 118 . The test runner 126 the tests 112 can be computed based upon data indicating the
executing on the test instances 124 can dequeue the tests 45 tests 112 that have already been executed , the estimated
from the queue 118 and perform the tests 112 on the software execution time for the tests 112 that remain to be executed ,
under test 116 . In the example shown in FIG . 1 , for instance, and the estimated execution time for tests 112 that are
the software testing service 104 has placed the tests 112A - currently executing . The test runner 126 can provide someor
112C on the test queue 118 . The test runner 126 executing all of this data to the software testing service 104 .
on the test instance 124 A has dequeued and executed the test 50 Using the data described above, the software testing
112A . Similarly, the test runner 126 executing on the test service 104 can also determine whether the estimated
instance 124B has dequeued and executed the test 112C . amount of time for completion of the tests 112 is greater than
Likewise , the test runner 126 executing on the test instance or less than a desired ( e .g . an amount of time specified by the
124C has dequeued and executed the test 112C . The test software developer 106 ) amount of time for completing the
runner 126 will continue dequeueing tests 112 and executing 55 tests 112 . If the estimated time for completion of the tests
the tests 112 until no further tests 112 (or references to tests) 112 is less than the desired amount of time for completion
remain on the queue 118 . (i. e . the tests 112 will complete sooner than desired ), then
The test runner 126 is also configured to store test results one or more of the test instances 124 can be de -provisioned
128 in an appropriate location , such as a database , a network by sending a message to the on - demand computing service
storage location , another queue, or in another appropriate 60 122 .
data store . For instance , in the example shown in FIG . 1, the If,however, the estimated time for completion of the tests
test runner 126 executing on the test instance 124A has 112 is greater than the desired amount of time for completion
generated and stored the test result 128A , the test runner 126 (i.e. the tests 112 will complete later than desired ), then
executing on the test instance 124B has generated and stored additional test instances 124 can be provisioned and con
the test result 128B , and the test runner 126 executing on the 65 figured in the manner described above. In this way, the
test instance 124C has generated the test result 128C . The number of test instances 124 can be dynamically optimized
software developer 106 can utilize the developer computing during testing in order to complete a suite of tests 112
US 9 ,811 ,451 B1
approximately within a specified amount of time. Additional desired completion time for the tests 112 , and an estimated
details regarding the mechanism described above for time for completion of the tests 112 . Other heuristics might
dynamically adjusting the number of test instances 124 also or alternately be utilized in other configurations . Addi
during test execution will be provided below with regard to tional details regarding this configuration of the software
FIG . 5 . 5 testing service 104 will be provided below with regard to
In some configurations, the test runner 126 can be con - FIGS . 6 and 7 .
figured to execute on multiple processor threads on each test It should be appreciated that the on - demand computing
instance 124 . In this implementation , the test runner 126 service 122 can provide different instance types for instan
executing on each thread can independently dequeue and tiating the test instances 124 . For example , and without
execute tests 112 . For instance , if the test runner 126 is 10 limitation , the test instances 124 provided by the service
executed on N threads on a test instance 124 , then N tests provider network 102 may be on -demand instances ,
112 can be dequeued and executed at a time. Similarly, reserved instances, spot instances, or standby instances
multiple test runners 126 can be executed in different 1010. Other types of instances not specifically described
processes. For example , M processes can be spawned to herein might also be utilized to implement the functionality
thereby perform M tests 112 concurrently . Both multithread - 15 disclosed herein .
ing and multi -process execution can also be utilized con - O -demand instances, which might also be referred to
currently, to thereby perform M * N tests 112 concurrently. herein as “ regular” instances, are instances that are paid for
Other configurations can also be utilized . and in active use . On - demand instances allow customers of
In other configurations, the software testing service 104 the service provider network 102 , such as the developer 109 ,
can also receive metrics from the test runner 126 . For 20 to pay for capacity per unit of time, such as an instance -hour,
example, the test runner 126 can provide information regard without requiring a long- term commitment. This frees the
ing the CPU usage , memory usage , thread count, and /or customer from the costs and complexities of planning ,
other metrics for the test instances 124 . Based on this purchasing, and maintaining hardware and transformswhat
information , the software testing service 104 can instruct the are commonly large fixed costs into much smaller variable
test runner 126 to dynamically adjust the concurrent execu - 25 costs .
tion of the test runner 126 ( e . g . multithreaded or multi Reserved instances are instances of a computing resource
process execution as described above) based upon whether that are reserved in exchange for a payment. Reserved
the test instances 124 are being under -utilized or not. Addi- instances provide the customer the option to make a one
tionally, in other configurations this information can be time payment for each instance they want to reserve . In turn ,
utilized to select different types of test instances 124 based 30 the customer may receive a significant discount on the
upon the execution of the tests 112 . For example , and hourly usage charge for the reserved instances as compared
without limitation , a test instance 126 with a greater amount to on -demand instances. After a customer makes a one-time
of memory might be selected for future execution ofthe tests payment for the reserved instances, the reserved instances
112 if the tests 112 are memory bound . Similarly, a test are reserved for the customer and the customer has no
instance 126 with a higher capacity CPU might be selected 35 further obligation . The customer may choose to run the
for future execution of the tests 112 if the tests 112 are CPU reserved instances for the discounted usage rate for the
bound . Likewise , a test instance 126 with a greater amount duration of a chosen term . If the customer does not use the
of network bandwidth might be selected for future execution reserved instances, the customer will not pay usage charges
of the tests 112 if the tests 112 are bandwidth bound . Other on these instances.
mechanisms can also be utilized to select different types of 40 Spot instances allow customers to bid on unused capacity
tests instances 124 for performing the tests 112 . in the on - demand computing network 122 . The customer can
In yet other configurations, the software testing service run the spot instances for as long as their bid exceeds a
104 is configured to operate in conjunction with a static fleet current market price , which might be referred to herein as
of test instances 124. In this configuration , the number of the “ spot instance market price .” The spot instance market
available test instances 124 is fixed and cannot be modified 45 price can fluctuate based upon supply and demand . Spot
dynamically as in the configuration described above . Addi- instances may be terminated if a customer ' s maximum bid
tionally, in this configuration the software testing service no longer exceeds the current spot instance market price .
104 can be configured to distribute tests 112 for multiple In order to obtain spot instances, the customer places a
different service clients ( e .g . customers ) to the fixed number request for the spot instances that specifies the desired
of test instances 112 . For example , and without limitation , 50 number of spot instances and the maximum price the cus
the software testing service 104 might receive requests from tomer is willing to pay per instance hour. If the customer 's
multiple service clients (not shown in FIG . 1 ) to perform maximum price bid exceeds the current spot instance market
tests on different software 116 . price for the spot instances , the customer ' s request will be
Utilizing one or more heuristics , the software testing fulfilled and the customer 's spot instances will run until
service 104 can select a subset of the tests 112 for one of the 55 either the customer chooses to terminate them or the market
service clients and a subset of the tests 112 for another of the price increases above the customer 's maximum price
service clients. The selected subsets of the tests 112 can then (whichever is sooner ). Various components operating within
be made available to a test runner 126 executing on the test the on -demand computing service 122 and /or the service
instances 124 in the static fleet . Once the tests 112 have been provider network 102 may manage the market for the spot
completed , the heuristicsmight be utilized to select another 60 instances, including setting the current spot instance market
subset of the tests 112 , which are then made available to the price for the spot instances .
test runners 126 and performed on the software 116 under Standby instances are spot instances that have been
test. This process can continue until no further tests 112 acquired on behalf of a customer and that are made ready for
remain to be performed for any of the clients . near immediate use by the customer in the manner described
Theheuristics utilized to select subsets of the tests 112 for 65 herein . The price charged for the standby instances is
performance on the software 116 under test can include, but typically less than the price charged for the on -demand
are not limited to , a priority associated with the tests 112 , a instances, since the standby instances may be terminated in
US 9,811, 451 B1
13 14
the samemanner as the spot instances . In one configuration , the test instances 124 . Once the test instances 124 have been
the standby instances are priced higher than the spot provisioned , the routine 200 proceeds to operation 208 ,
instances and the reserved instances, but lower than the where the test runner 126 and the software 116 under test are
on -demand instances. It should be appreciated , however, deployed to the test instances 124 . Other services, such as a
that the various pricing mechanismsdescribed above for the 5 software deployment service , executing within the service
on - demand instances, reserved instances, spot instances, and provider network 102 can be utilized to perform the deploy
standby instances are merely illustrative and that other ment of the test runner 126 and the software 116 under test.
mechanismsmay be utilized to set the pricing for the various From operation 208 , the routine 200 proceeds to operation
instance types . 210 , where the software testing service 104 places the tests
It should be appreciated that, in various configurations , 10 112 on the test queue 118 in one particular configuration . In
the software testing service 104 can utilize various combi - other configurations , the tests 112 might be placed in another
nations of the instance types described above to execute the type of location , such as within a database or in a network
tests 112 . For example , and without limitation , the software storage location , for retrievalby the test runner 126 . In other
testing service 104 can execute tests on the various instance configurations, the tests 112 can be placed on the test queue
types in order to benchmark aspects of their performance . 15 118 prior to or in parallel with the provisioning of the test
Based upon these benchmarks, the most appropriate instance instances 124 . In other configurations, the tests 112 can be
type , or types, for executing certain tests 112 can be selected . placed on the test queue 118 and dequeued at a future point
Other mechanisms can also be utilized to select a particular in time based upon the spot instance market price , which
instance type for executing a particular test 112 . was described above . Other configurations can also be
FIG . 2 is a flow diagram showing a routine 200 that 20 utilized .
illustrates aspects of the operation of the software testing As discussed above, and in further detail below with
service 104 shown in FIG . 1, according to one particular regard to FIG . 4 , the test runner 126 dequeues and executes
configuration disclosed herein . It should be appreciated that the tests 112 . The test runner 126 then places the test results
the logical operations described herein with respect to FIG . 128 in an appropriate storage location for retrieval by the
2 , and the other FIGS., can be implemented ( 1 ) as a sequence 25 software developer 106 and the software testing service 104 .
of computer implemented acts or program modules running As discussed above, the test results 128 can identify the
on a computing system and/or ( 2) as interconnected machine particular test 112 that was executed and describe the
logic circuits or circuit modules within the computing sys amount of time required to perform the test 112 . The test
tem . results 128 can also include other types of data .
The implementation of the various components described 30 From operation 210 , the routine 200 proceeds to operation
herein is a matter of choice dependent on the performance 212 , where the software testing service 104 receives the test
and other requirements of the computing system . Accord results 128 from the test runner 126 . As discussed above, the
ingly , the logical operations described herein are referred to test results 128 can include data indicating the time it took
variously as operations , structural devices, acts, or modules. to execute the tests 112 . This information is utilized to
These operations, structural devices , acts , and modules can 35 update the historical test execution time data 126 at opera
be implemented in software , in firmware , in special purpose tion 214 . For example , and without limitation , data indicat
digital logic, and any combination thereof. It should also be i ng the average execution time for a particular test 112 can
appreciated that more or fewer operations can be performed be generated and stored .
than shown in the FIGS. and described herein . These opera - From operation 214 , the routine 200 proceeds to operation
tions can also be performed in parallel, or in a different order 40 216 , where the software testing service 104 determines
than those described herein . Some or all of these operations whether all of the requested tests 112 have completed . If the
might also be performed by components other than those tests 112 have not yet completed , the routine 200 proceeds
specifically identified . back to operation 212 , where addition test results 128 can be
The routine 200 begins at operation 202 , where the received and the corresponding historical test execution time
software testing service 104 receives a test request 110 . As 45 data 126 can be updated . If, however, all of the tests 112
discussed above , the test request 110 can include or provide have been completed , the routine 200 proceeds from opera
a reference to the tests 112, include or provide a reference to tion 216 to operation 218 .
the software 116 under test, and specify the testing prefer - At operation 218 , the software testing service 104 causes
ences 112 . As also discussed above , the testing preferences the test instances 124 that were provisioned at operation 206
112 can explicitly specify the number of test instances 124 50 to be de -provisioned . For example , and without limitation ,
to utilize for performing the tests 112 on the software 116 the software testing service 104 can transmit an instruction
under test. As another example , the testing preferences to the on -demand computing service 122 via an appropriate
might specify a maximum execution time for the tests 112 . network services API to de -provision the test instances 124 .
The testing preferences 114 can provide data defining other Once the test instances 124 have been de - provisioned , the
aspects of the manner in which the tests 112 are to be 55 routine 300 proceeds from operation 218 to operation 220 ,
executed in other configurations . where it ends.
From operation 202 , the routine 200 proceeds to operation FIG . 3 is a flow diagram showing a routine 300 that
204 , where the software testing service 104 determines the illustrates aspects of the operation of the software testing
number of tests instances 124 to utilize for performing the service 104 shown in FIG . 1 for determining a number of test
tests 112 . Additional details regarding this process will be 60 instances 124 to utilize for testing, according to one par
provided below with regard to FIG . 3. From operation 202, ticular configuration disclosed herein . The routine 300
the routine 200 proceeds to operation 206 . begins at operation 302, where the software testing service
At operation 206 , the software testing service 104 causes 104 determines whether the testing preferences 114 specify
the number of test instances 124 determined at operation 204 the number of test instances 124 to use for testing . If so , the
to be provisioned . For instance , in one particular configu - 65 routine 300 proceeds from operation 302 to operation 302 ,
ration , the software testing service 104 transmits an instruc- where the number of test instances 124 specified in the
tion to the on -demand computing service 122 to provision testing preferences 114 are utilized for testing . The routine
US 9,811, 451 B1
15 16
300 then proceeds from operation 304 , to operation 320 . At tests 112 to compute an estimate of the time to complete the
operation 320 , the routine 300 returns to operation 206 , requested tests 112 . As discussed above, in some configu
described above with regard to FIG . 2 . rations historical execution times for tests 112 in the same
If, at operation 302 , the software testing service 104 class as a requested tests 112 might be given a greater weight
determines that the number of test instances 124 to utilize for 5 than a test 112 in the same package . Similarly , historical
testing is not specified in the testing preferences 112 , the execution times for tests 112 in the same package might be
routine 300 proceeds from operation 302 to operation 306 . assigned a greater weight than the historical execution times
At operation 306 , the software testing service 104 deter - of tests 112 in the same archive . In this manner, the historical
mines whether the testing preferences 114 specify the maxi - execution times for other tests 112 can be assigned a weight
mum execution time for the tests 112 . If the testing prefer - 10 that is based upon their relevance to the tests 112 to be
ences 114 do not specify the number of test instances 124 or performed . Other mechanisms can also be utilized to esti
the maximum testing time, the routine 300 proceeds from mate the time to complete the tests 112 from historical
operation 306 to operation 308 , where a default number of execution times of other tests . From operation 318 , the
test instances 124 can be utilized. When a default number of routine 300 proceeds to operation 314 , where the number of
test instances 124 are utilized , the number of test instances 15 test instances 124 to utilize can be computed based upon the
124 can be dynamically increased and /or decreased in the estimated completion time computed at operation 318 . Pro
manner described below with regard to FIG . 5 . In this way, cessing then proceeds from operation 314 to operation 206 ,
the software developer 106 can be freed from having to described above with regard to FIG . 2 .
specify a number of test instances 124 or the maximum FIG . 4 is a flow diagram showing a routine 400 that
execution time. 20 illustrates aspects of the operation of the test runner 126
If , at operation 306 , the software testing service 104 illustrated in FIG . 1 , according to one particular configura
determines that the testing preferences 114 do specify a tion disclosed herein . The routine 400 begins at operation
maximum execution time for the tests 112 , the routine 300 402 , where the test runner 126 executing on a test instance
proceeds from operation 306 to operation 310 . At operation 124 polls the queue 118 for a test 112 . If the queue 118 does
310 , the software testing service 104 determines whether 25 not contain any tests 112 , the routine 400 proceeds to
historical test execution time data 126 is available for the operation 406 , where the test runner 126 may wait a speci
tests 112 to be performed . If so , the routine 300 proceeds fied period of time before again polling the queue 118 . If a
from operation 310 to operation 312 . test 112 is present on the queue 118 , the routine 400 proceeds
At operation 312 , the software testing service 104 com from operation 404 to operation 408 .
putes an estimate of the time to complete the tests 112 based 30 At operation 408 , the test runner 126 dequeues a test 112
upon the historical test execution time data 126 for the tests from the test queue 118 . The routine 400 then proceeds to
112 . As discussed above , in one particular configuration a operation 410 , where the test runner 126 performs the
window of N executions of a test 112 might be utilized , and dequeued test 112 on the software 116 under test. As
the execution times for more recent executions of the test discussed briefly above, the test 112 can be a unit test, an
112 will be assigned a higher weight than older executions . 35 integration test, a performance test, or another type of
Other mechanisms might also be utilized to compute an software test. Once the test 112 has been performed , the
estimate of the execution time for the tests 112 based upon routine 400 proceeds to operation 412 , where the test runner
previous executions of the same tests 112 . 126 generates a test result 128 for the test 112 , including data
140
From operation 312 , the routine 300 proceeds to operation indicating the time required to perform the test 112 . From
314 , where the estimate of the execution time computed at 40 operation 412 , the routine 400 proceeds back to operation
operation 312 is utilized to compute the number of test 402 , where another test 112 can be dequeued and performed
instances 124 need to perform the requested tests 112 within in the manner described above .
the maximum execution time specified in the testing pref- FIG . 5 is a flow diagram showing a routine 500 that
erences 114 . For example , and without limitation , if the illustrates aspects of the operation of the software testing
software testing service 104 determines that the tests 112 45 service 104 shown in FIG . 1 for dynamically adjusting the
will take three hours to complete , the software testing number of test instances 124 used for testing during test
service
se 104 will compute the number of instances 124 execution , according to one particular configuration dis
required to complete the tests 112 in three hours if executed closed herein . The routine 500 begins at operation 502 ,
in parallel. Once the number of instances 124 has been where the software testing service 104 obtains data identi
computed at operation 314 , processing returns to operation 50 fying the tests 112 that have been completed , the tests 112
206 , described above with regard to FIG . 2 . that are currently executing , the tests 112 that have not yet
If, at operation 310 , the software testing service 104 executed , the elapsed time since execution of the tests 112
determines that historical test execution time data 126 is not started , and a desired duration for completion of the tests
available for the tests 112 to be executed , the routine 300 112 . As mentioned above , the test request 110 can specify
proceeds from operation 310 to operation 316 . At operation 55 the desired duration .
316 , the software testing service 104 determines whether From operation 502 , the routine 500 proceeds to operation
historical test execution time data 126 exists for tests 112 504 , where the software testing service 104 utilizes the data
that are similar to the tests 112 to be executed . As discussed identifying the tests 112 that have been completed , the tests
above, similar tests can include, but are not limited to , tests 112 that are currently executing, the tests 112 that have not
112 in the same class, archive , or package . If historical test 60 yet executed , the elapsed time since execution of the tests
execution time data 126 exists for similar tests 112 , the 112 started , and the historical test execution time data 126 to
routine 300 proceeds from operation 316 to operation 318 . compute an estimate of the amount of timeneed to complete
If not, the routine 300 proceeds from operation 316 to all of the requested tests 112 . Once this value has been
operation 308 , where a default number of test instances 124 computed , the routine 500 proceeds from operation 504 to
can be utilized . 65 operation 506 .
Atoperation 318 , the software testing service 104 utilizes At operation 506 , the software testing service 104 deter
the historical test execution time data 126 for the similar mines whether the estimated amount of time to complete all
US 9,811,451 B1
18
of the tests 112 is less than the desired testing duration (i.e . 112B , and 112C . The selected subsets of the tests 112 can
that the tests 112 will complete sooner than desired ). If the then be made available to a test runner 126 executing on the
tests 112 will complete sooner than desired , the routine 500 test instances 124 in the static fleet using a queue 118 as in
proceeds from operation 506 to operation 508 . At operation the configuration described above.
508 , the software testing service 104 can cause one or more 5 Once the subsets of the tests 112 placed on the queue 118
of the test instances 124 to be de -provisioned . For example , have been completed , the software testing service 104 can
the software testing service 104 can instruct the on - demand utilize the heuristics select another subset of the tests 112 to
computing service 122 to de -provision one or more of the be performed ( e .g . the tests 112A , 112B , and 112C in FIG .
test instances 124 . By reducing the number of test instances 6 ), which are then made available to the test runners 126 and
124 in this manner, the amount of time to perform the 10 performed on the software 116 under test in the same
requested tests 112 can be increased so that the total amount manner . This process can continue until no further tests 112
of time to perform all of the tests 112 is approximately the remain to be performed for any of the requesting clients.
desired duration . The heuristics utilized by the software testing service 104
If, at operation 506 , the software testing service 104 to select subsets of the tests 112 requested by different
determines that the tests 112 will not complete sooner than 15 service clients for performance on the software 116 under
desired , the routine 500 proceeds to operation 510 . At t est using a static fleet 602 of test instances 124 can include ,
operation 510 , the software testing service 104 determines but are not limited to , a priority associated with each of the
whether the estimated amount of time to complete all of the tests 112 , a desired completion time for the tests 112, and an
tests 112 is greater than the desired testing duration (i. e . that estimated time for completion of the tests 112 . Other heu
the tests 112 will complete later than desired ). If the tests 112 20 ristics might also or alternately be utilized in other configu
will not complete sooner or later than desired , the routine rations.
500 proceeds back to operation 502 , where the process Using some or all of the heuristics described above , the
described above can be repeated . If, however , performance software testing service 104 can make a best attempt to
of the tests 112 will complete later than desired , the routine satisfy the various execution times or other priorities
500 proceeds from operation 510 to operation 512 . 25 requested by the different service clients . For example , using
At operation 512, the software testing service 104 can this mechanism tests 112 having a higher priority can be
cause one or more additional test instances 124 to be given preferential treatment in the test queue 118 . Similarly,
provisioned . For example , the software testing service 104 the priority values can be utilized to resolve conflicts
can instruct the on -demand computing service 122 to pro - between test requests submitted by different service clients .
vision one or more additional test instances 124 . By increas - 30 Tests 112 having shorter execution times or other attributes
ing the number of test instances 124 in this manner, the might also be given preferential treatment in other configu
amount of time to perform the requested tests 112 can be rations. Additional details regarding the configuration of the
reduced so that the total amount of time to perform all of the software testing service 104 shown in FIG . 6 are provided
tests 112 is approximately the desired duration . From opera below with regard to FIG . 7 .
tion 512 , the routine 500 proceeds back to operation 502, 35 FIG . 7 is a flow diagram showing a routine 700 that
where the process described above can be repeated . illustrates aspects of the operation of the software testing
FIG . 6 is a system architecture diagram showing another service 104 for distribution of tests 126 for multiple clients
configuration of the software testing service 104 in which to a shared static fleet 602 of test instances 124 . The routine
tests 112 for multiple clients of the software testing service 700 begins at operation 702 , where the software testing
104 are distributed to a shared static fleet 602 of test 40 service 104 receives tests from multiple service clients . For
instances 124 . As shown in FIG . 6 and described briefly example , and as shown in FIG . 6 , the software testing
above , the software testing service 104 can be configured to service 104 can receive tests 112A , 112B , and 112C from
operate in conjunction with a static fleet 602 of test instances multiple developer computing devices 108A , 108B , and
124 . In this configuration , the number of available test 108C , respectively . Each of the received tests 112 can have
instances 124 is fixed and cannot be modified dynamically 45 an associated software 116 under test ( e . g . the software
as in the configuration described above . under test 116A , 116B , and 116C ) . Each of the received tests
Additionally, in this configuration the software testing 112 can also have associated testing preferences 114 ( e . g . the
service 104 can be configured to distribute tests 112 for testing preferences 114A , 114B , and 114C ).
multiple different service clients ( e .g . customers ) to the fixed From operation 702, the routine 700 proceeds to operation
number of test instances 112 . For example , and without 50 704 , where the software testing service 104 can utilize
limitation , the software testing service 104 might receive a various heuristics to select subsets of the requested tests 112 .
request 110A from a developer computing device 108A to For instance , in the example shown in FIG . 6 , the software
perform tests 112 A on software under test 116A , a request testing service 104 can utilize the heuristics to select a subset
110B from a developer computing device 108B to perform of the tests 112A , 112B , and 112C to place on the test queue
tests 112B on software under test 116B , and a request 110C 55 118 . As discussed above , the heuristics utilized by the
from a developer computing device 108C to perform tests software testing service 104 to select subsets of the tests 112
112A on software under test 116C . Additionally , each of the requested by different service clients for performance on the
test requests 110 might specify separate testing preferences software 116 under test using a static fleet 602 of test
114 . For instance , the testing preferences 114A might instances 124 can include , but are not limited to , a priority
request that the tests 112 A be performed within a specified 60 associated with each of the tests 112 , a desired completion
amount of time. Similarly , the testing preferences 114B and time for the tests 112 , and an estimated time for completion
114C might specify that the tests 112B and 112C be per- of the tests 112 . Other heuristics might also or alternately be
formed in different amounts of time, respectively. utilized in other configurations . Using some or all of these
Utilizing one or more heuristics, the software testing heuristics, the software testing service 104 can make a best
service 104 can select a subset of the tests 112 for each of 65 attempt to satisfy the various execution times or other
the requesting service clients. For example , the software priorities requested by the different service clients in the
testing service 104 can select a subset of the tests 112A , testing preferences 114 .
US 9,811, 451 B1
19 20
From operation 704 , the routine 700 proceeds to operation by the service provider network 102 over a network 802 ,
706 , where the software testing service 104 places the which can be a wide area communication network (“ WAN ” ),
subsets of the tests 112 selected at operation 704 on the test such as the Internet, an intranet or an Internet service
queue 118 . As in the configurations described above, the test provider (“ ISP ” ) network or a combination of such net
runner 126 executing on the test instances 124 in the static 5 works . For example , and without limitation , a computing
fleet 602 can dequeue the tests 112 and perform the tests on device 800 operated by a customer or other user of the
the various software 116 under test. service provider network 102 , such as the developer com
From operation 706 , the routine 700 proceeds to operation puting device 108 , might be utilized to access the service
708 , where the software testing service 104 determines if provider network 102 by way of the network 802 . It should
more tests 112 remain to be executed . If so , the routine 700 10 be appreciated that a local- area network (“ LAN ” ), the Inter
proceeds from operation 708 to operation 710 , where the net, or any other networking topology known in the art that
software testing service 104 may wait for a period of time connects the data centers 804 to remote customers and other
( e . g . one minute ) prior to selecting the next subset of the users can be utilized . It should also be appreciated that
requested tests 112 . combinations of such networks might also be utilized .
Once the period of time has elapsed , the routine 700 15 FIG . 9 is a computing system diagram that illustrates one
proceeds from operation 710 to operation 704 , where addi- configuration for a data center 804 that implements aspects
tional tests 112 can be processed in the manner described of the technologies disclosed herein for distributed software
above. This process can continue until no further tests 112 testing. The example data center 804 shown in FIG . 9
remain to be performed for any of the requesting clients . includes several server computers 902A - 902F (which might
Once no further tests 112 remain to be performed , the 20 be referred to herein singularly as “ a server computer 902 ”
routine 700 proceeds from operation 708 to operation 712 , or in the plural as “ the server computers 902" ) for providing
where it ends. the computing resources 904A -904E .
FIG . 8 is a system and network diagram that shows one The server computers 902 can be standard tower, rack
illustrative operating environment for the configurations mount, or blade server computers configured appropriately
disclosed herein that includes a service provider network 25 for providing the computing resources 904 described herein
102 that can be configured to provide functionality for illustrated in FIG . 9 as the computing resources 904A
distributed software testing in the manner described above , 904E ). As mentioned above , the computing resources 904
according to one configuration disclosed herein . As dis provided by the service provider network 102 might be data
cussed above , the service provider network 102 can execute processing resources such as VM instances or hardware
network services that provide computing resources on a 30 computing systems, data storage resources, database
permanent or an as -needed basis . Among other types of resources, networking resources , and others . Some of the
functionality , the computing resources provided by the ser - servers 902 might also be configured to execute network
vice provider network 102 can be utilized to implement the services 906A - 906E , respectively, capable of instantiating,
various services described herein . As also discussed above , providing and /or managing the computing resources 904 ,
the computing resources provided by the service provider 35 some of which are described in detail below with regard to
network 102 can include various types of computing FIG . 10 .
resources , such as data processing resources like VM The data center 804 shown in FIG . 9 also includes a server
instances, data storage resources, networking resources, data computer 902F that can execute some or all of the software
communication resources , network services, and the like. components described above . For example , and without
Each type of computing resource provided by the service 40 limitation, the server computer 902F might be configured to
provider network 102 can be general- purpose or can be execute the software testing service 104 , which was
available in a number of specific configurations . For described in detail above . The server computer 902F might
example , data processing resources can be available as also be configured to execute other components and / or to
physical computers or VM instances in a number of different store data for providing some or all of the functionality
configurations. The VM instances can be configured to 45 described herein . In this regard , it should be appreciated that
execute applications , including Web servers , application the software testing service 104 might execute on many
servers , media servers , database servers, some or all of the other physical or virtual servers in the data centers 804 in
services described above, and / or other types of software . various configurations.
Data storage resources can include file storage devices, In the example data center 804 shown in FIG . 9 , an
block storage devices , and the like . The service provider 50 appropriate LAN 908 is also utilized to interconnect the
network 102 might also be configured to provide other types server computers 902A - 902F . The LAN 908 is also con
of computing resources not mentioned specifically herein . nected to the network 802 illustrated in FIG . 8 . It should be
As also discussed above, the computing resources 108 appreciated that the configuration and network topology
provided by the service provider network 102 are enabled in described herein has been greatly simplified and that many
one implementation by one or more data centers 804A - 804D 55 more computing systems, software components, networks ,
(which might be referred herein singularly as “ a data center and networking devices can be utilized to interconnect the
804” or in the plural as “ the data centers 804” ). The data various computing systems disclosed herein and to provide
centers 804 are facilities utilized to house and operate the functionality described above . Appropriate load balanc
computer systems and associated components . The data ing devices or other types ofnetwork infrastructure compo
centers 804 typically include redundant and backup power, 60 nents might also be utilized for balancing a load between
communications, cooling, and security systems. The data each of the data centers 804A - 804D , between each of the
centers 804 can also be located in geographically disparate server computers 902A - 902F in each data center 804 , and ,
locations. One illustrative configuration for a data center 804 potentially , between computing resources 904 in each of the
that might be utilized to implement the technologies dis - data centers 804 . It should be appreciated that the configu
closed herein will be described below with regard to FIG . 9 . 65 ration of the data center 804 described with reference to FIG .
The customers and other users of the service provider 9 is merely illustrative and that other implementations might
network 102 can access the computing resources provided be utilized .
US 9 ,811,451 B1
22
FIG . 10 is a system and network diagram that shows The storage service 906F might include software and
aspects of several network services that might be provided computing resources that collectively operate to store data
by and utilized within a service provider network 102 in one using block or file -level storage devices (and/or virtualiza
configuration disclosed herein . In particular, and as dis - tions thereof). The storage devices of the storage service
cussed above, the service provider network 102 can provide 5 906F might, for instance , be operationally attached to virtual
a variety of network services to customers and other users of computer systems provided by the on -demand computing
the service provider network 102 including, but not limited service 122 to serve as logical units (e . g ., virtual drives ) for
to , the software testing service 104 , the queue service 120 , the computer systems. A storage device might also enable
and the on -demand computing service 122 . The service the persistent storage of data used / generated by a corre
provider network 102 might also provide other types of 10 sponding virtual computer system where the virtual com
services including, butnot limited to , a storage service 906F, puter system service might only provide ephemeral data
a deployment service 906G , a cryptography service 906H , storage .
an authentication service 9061, and/or a policy management The service provider network 102 can also include a
service 906 ), each of which is described in greater detail cryptography service 906H . The cryptography service 906H
below . Additionally, the service provider network 102 might 15 can utilize storage services of the service provider network
also provide other services, some of which are described in 102 , such as the storage service 906F, to store encryption
greater detail below . keys in encrypted form , whereby the keys can be usable to
It should be appreciated that customers of the service decrypt customer keys accessible only to particular devices
provider network 102 can include organizations or individu of the cryptography service 906H . The cryptography service
als that utilize some or all of the services provided by the 20 906H might also provide other types of functionality not
service provider network 102. As described above , a cus - specifically mentioned herein .
tomer or other user can communicate with the service As illustrated in FIG . 10 , the service provider network
provider network 102 through a network , such as the net . 102 , in various configurations, also includes an authentica
work 802 shown in FIG . 8 . Communications from a cus- tion service 9061 and a policy management service 906J.
tomer computing device , such as the developer computing 25 The authentication service 9061, in one example , is a com
device 108 shown in FIG . 1, to the service provider network puter system (i.e ., collection of computing resources 904 )
102 can cause the services provided by the service provider configured to perform operations involved in authentication
network 102 to operate in accordance with the described of users . For instance , one of the services shown in FIG . 10
configurations or variations thereof. can provide information from a user to the authentication
It is noted that not all configurations described include the 30 service 9061 to receive information in return that indicates
services described with reference to FIG . 10 and that addi whether or not the requests submitted by the user are
tional network services can be provided in addition to or as authentic .
an alternative to the services explicitly described herein . The policy management service 906 ) , in one example , is
Each of the services shown in FIG . 10 might also expose a network service configured to manage policies on behalf of
Web service interfaces that enable a caller to submit appro - 35 customers or internal users of the service provider network
priately configured application programming interface 102 . The policy management service 906J can include an
(" API" ) calls to the various services through Web service interface that enables customers to submit requests related to
requests . In addition , each of the services might include the management of policy . Such requests can , for instance ,
service interfaces that enable the services to access each be requests to add, delete , change or otherwise modify
other ( e. g ., to enable a virtual computer system provided by 40 policy for a customer service, or system , or for other
the on -demand computing service 122 to store data in or administrative actions, such as providing an inventory of
retrieve data from the data storage service 906F ). Additional existing policies and the like .
details regarding some of the services shown in FIG . 10 will The service provider network 102 can additionally main
now be provided . tain other services based , at least in part, on the needs of its
As discussed above , the on - demand computing service 45 customers . For instance , the service provider network 102
122 can be a collection of computing resources configured can maintain a deployment service 906G for deploying
to instantiate VM instances and to provide other types of software and/ or a database service (not shown in FIG . 10 ) in
computing resources 904 on demand . For example , a cus- some configurations. A database service can be a collection
tomer or other user of the service provider network 102 can of computing resources that collectively operate to create ,
interact with the on -demand computing service 122 ( via 50 maintain , and allow queries to be performed on databases
appropriately configured and authenticated API calls ) to stored within the service provider network 102. For
provision and operate VM instances that are instantiated on example, a customer or other user of the service provider
physical computing devices hosted and operated by the network 102 can operate and manage a database from the
service provider network 102 . The VM instances can be used database service by utilizing appropriately configured net
for various purposes, such as to operate as servers support- 55 work API calls. This, in turn , can allow the customer to
ing a Web site, to operate business applications or, generally , maintain and potentially scale the operations in the database .
to serve as computing resources for the customer. Other Other services include object-level archival data storage
applications for the VM instances can be to support database services, and services that manage, monitor , interact with , or
applications , electronic commerce applications, business support other services . The service provider network 102
applications and /or other applications. Although the on - 60 might also be configured with other network services not
demand computing service 122 is shown in FIG . 10 , any specifically mentioned herein in other configurations .
other computer system or computer system service can be FIG . 11 shows an example computer architecture for a
utilized in the service provider network 102 , such as a computer 1100 capable of executing software components
computer system or computer system service that does not for implementing the functionality described above. The
employ virtualization and instead provisions computing 65 computer architecture shown in FIG . 11 illustrates a con
resources on dedicated or shared computers /servers and /or ventional server computer, workstation , desktop computer,
other physical devices . laptop, tablet, network appliance , e-reader, smartphone, or
US 9 ,811,451 B1
23 24
other computing device , and can be utilized to execute any tion . Examples of such factors can include, but are not
of the software components presented herein . limited to , the technology used to implement the physical
The computer 1100 includes a baseboard 1102, or “ moth - storage units, whether the mass storage device 1118 is
erboard,” which is a printed circuit board to which a characterized as primary or secondary storage, and the like .
multitude of components or devices can be connected by 5 For example , the computer 1100 can store information to
way of a system bus or other electrical communication the mass storage device 1118 by issuing instructions through
paths. In one illustrative configuration , one or more central the storage controller 1114 to alter the magnetic character
processing units (“ CPUs" ) 1104 operate in conjunction with istics of a particular location within a magnetic disk drive
a chipset 1106 . The CPUs 1104 can be standard program - unit, the reflective or refractive characteristics of a particular
mable processors that perform arithmetic and logical opera - 10 location in an optical storage unit , or the electrical charac
tions necessary for the operation of the computer 1100 . teristics of a particular capacitor, transistor, or other discrete
The CPUs 1104 perform operations by transitioning from component in a solid -state storage unit. Other transforma
one discrete , physical state to the next through the manipu - tions of physicalmedia are possible without departing from
lation of switching elements that differentiate between and the scope and spirit of the present description , with the
change these states. Switching elements can generally 15 foregoing examples provided only to facilitate this descrip
include electronic circuits that maintain one of two binary tion . The computer 1100 can further read information from
states , such as flip - flops, and electronic circuits that provide the mass storage device 1118 by detecting the physical states
an output state based on the logical combination of the states or characteristics of one or more particular locations within
ofone or more other switching elements, such as logic gates. the physical storage units .
These basic switching elements can be combined to create 20 In addition to the mass storage device 1118 described
more complex logic circuits, including registers, adders above, the computer 1100 can have access to other com
subtractors , arithmetic logic units , floating -point units, and puter - readable storage media to store and retrieve informa
the like. tion , such as program modules, data structures, or other data .
The chipset 1106 provides an interface between the CPUS It should be appreciated by those skilled in the art that
1104 and the remainder of the components and devices on 25 computer -readable storage media is any available media that
the baseboard 1102 . The chipset 1106 can provide an inter - provides for the non -transitory storage of data and that can
face to a RAM 1108 , used as the main memory in the be accessed by the computer 1100 .
computer 1100 . The chipset 1106 can further provide an By way of example , and not limitation, computer-read
interface to a computer -readable storage medium such as a able storage media can include volatile and non - volatile ,
read -only memory (“ ROM " ) 1110 or non -volatile RAM 30 removable and non -removable media implemented in any
(“ NVRAM ” ) for storing basic routines that help to startup method or technology. Computer -readable storage media
the computer 1100 and to transfer information between the includes , but is not limited to , RAM , ROM , erasable pro
various components and devices. The ROM 1110 or grammable ROM (“ EPROM ” ), electrically -erasable pro
NVRAM can also store other software components neces grammable ROM (“ EEPROM ” ), flash memory or other
sary for the operation of the computer 1100 in accordance 35 solid -state memory technology , compact disc ROM (“ CD
with the configurations described herein . ROM ” ), digital versatile disk (“ DVD ” ), high definition
The computer 1100 can operate in a networked environ - DVD (“ HD -DVD ” ), BLU -RAY, or other optical storage ,
ment using logical connections to remote computing devices magnetic cassettes, magnetic tape, magnetic disk storage or
and computer systems through a network , such as the other magnetic storage devices, or any other medium that
network 908 . The chipset 1106 can include functionality for 40 can be used to store the desired information in a non
providing network connectivity through a NIC 1112, such as transitory fashion .
a gigabit Ethernet adapter. The NIC 1112 is capable of As mentioned briefly above , the mass storage device 1118
connecting the computer 1100 to other computing devices can store an operating system 1120 utilized to control the
over the network 908 . It should be appreciated thatmultiple operation of the computer 1100. According to one configu
NICs 1112 can be present in the computer 1100, connecting 45 ration , the operating system comprises the LINUX operating
the computer to other types of networks and remote com - system . According to another configuration , the operating
puter systems. system comprises the WINDOWS SERVER operating
The computer 1100 can be connected to a mass storage system from MICROSOFT Corporation . According to fur
device 1118 that provides non - volatile storage for the com - ther configurations, the operating system can comprise the
puter. The mass storage device 1118 can store an operating 50 UNIX operating system or one of its variants . It should be
system 1120 , programs 1122 , and data , which have been appreciated that other operating systems can also be utilized .
described in greater detail herein . The mass storage device The mass storage device 1118 can store other system or
1118 can be connected to the computer 1100 through a application programs and data utilized by the computer
storage controller 1114 connected to the chipset 1106 . The 1100 .
mass storage device 1118 can consist of one or more 55 In one configuration , the mass storage device 1118 or
physical storage units. The storage controller 1114 can other computer -readable storage media is encoded with
interface with the physical storage units through a serial computer -executable instructions which , when loaded into
attached SCSI (“ SAS” ) interface , a serial advanced technol - the computer 1100 , transform the computer from a general
ogy attachment (“ SATA ” ) interface, a fiber channel (“ FC ” ) purpose computing system into a special-purpose computer
interface , or other type of interface for physically connecting 60 capable of implementing the configurations described
and transferring data between computers and physical stor - herein . These computer -executable instructions transform
age units. the computer 1100 by specifying how the CPUs 1104
The computer 1100 can store data on the mass storage transition between states, as described above . According to
device 1118 by transforming the physical state of the physi- one configuration , the computer 1100 has access to com
cal storage units to reflect the information being stored . The 65 puter - readable storage media storing computer - executable
specific transformation of physical state can depend on instructions which , when executed by the computer 1100 ,
various factors , in different implementations of this descrip perform the various processes described above with regard
US 9,811, 451 B1
25 26
to FIGS . 1 -7 . The computer 1100 might also include com 2 . The apparatus of claim 1, wherein the testing prefer
puter-readable storage media for performing any of the other ences specify the number of test instances to utilize for
computer -implemented operations describeda herein
herein . performing the plurality of tests on the software .
The computer 1100 can also include one or more input/ 3. The apparatus of claim 1 , wherein the testing prefer
output controllers 1116 for receiving and processing input 5 ences specify a maximum execution time for the plurality of
from a number of input devices , such as a keyboard , a tests, and wherein the number of test instances to utilize for
mouse , a touchpad , a touch screen , an electronic stylus, or performing the plurality of tests on the software is deter
other type of input device . Similarly , an input/output con - mined based , at least in part, on historical test execution time
troller 1116 can provide output to a display, such as a data for the plurality of tests .
computer monitor , a flat -panel display, a digital projector, a 10 4 . The apparatus of claim 1 , wherein the testing prefer
printer, a plotter, or other type of output device . It will be ences specify a maximum execution time for the plurality of
appreciated that the computer 1100 might not include all of tests, and wherein the number of test instances to utilize for
the components shown in FIG . 11 , can include other com - performing the plurality of tests on the software is deter
ponents that are not explicitly shown in FIG . 11 , or might mined based , at least in part, on historical test execution time
utilize an architecture completely different than that shown 15 data for tests similar to the plurality of tests .
in FIG . 11 . 5 . The apparatus of claim 1 , wherein the non - transitory
Based on the foregoing , it should be appreciated that computer-readable storage medium has further instructions
technologies for distributed software testing have been pre - stored thereupon to :
sented herein . Moreover, although the subject matter pre compute an estimated amount of time for completion of
sented herein has been described in language specific to 20 the plurality of tests ;
computer structural features , methodological acts , and com determine that the estimated amount time for completion
puter readable media , it is to be understood that the inven of the plurality of tests is less than a desired amount of
tion defined in the appended claims is not necessarily limited time for completion of the plurality of tests ; and
to the specific features, acts , or media described herein . cause one or more of the test instances to be de-provi
Rather , the specific features , acts , and media are disclosed as 25 sioned by the on -demand computing service in
example forms of implementing the claims. response to determining that the estimated amount of
The subjectmatter described above is provided by way of time for completion of the plurality of tests is less than
illustration only and should not be construed as limiting . the desired amount of time for completion of the
Furthermore, the claimed subject matter is not limited to plurality of tests.
implementations that solve any or all disadvantages noted in 30 6 . The apparatus of claim 1 , wherein the non - transitory
any part of this disclosure . Various modifications and computer- readable storage medium has further instructions
changes can bemade to the subject matter described herein stored thereupon to :
without following the example configurations and applica compute an estimated amount of time for completion of
tions illustrated and described , and without departing from the plurality of tests;
the true spirit and scope of the present invention , which is set 35 determine that the estimated amount time for completion
forth in the following claims. of the plurality of tests is greater than a desired amount
What is claimed is : of time for completion of the plurality of tests ; and
1 . An apparatus , comprising : cause one or more additional test instances to be provi
a processor; and sioned by the on - demand computing service in
a non -transitory computer -readable storage medium hav - 40 response to determining that the estimated amount of
ing instructions stored thereupon which are executable time for completion of the plurality of tests is greater
by the processor and which , when executed , cause the than the desired amount of time for completion of the
apparatus to : plurality of tests.
receive a request to perform a plurality of tests on 7 . A computer-implemented method for performing a
software , the request comprising an indication of 45 plurality of tests on software, the method comprising :
testing preferences ; receiving a request to perform a plurality of tests on
determine, based at least in part on the testing prefer software ;
ences, a number of test instances to utilize for determining a number of test instances to utilize for
performing the plurality of tests on the software ; performing the plurality of tests on the software ;
responsive to determining the number of the test 50 responsive to determining the number of the test
instances , automatically cause the number of the test instances , automatically causing the number of test
instances to be provisioned by an on -demand com instances to be provisioned , wherein the test instances
puting service executing in a service provider net include one or more of a virtual machine , or a con
work , wherein the test instances include one or more tainer;
of a virtual machine , or a container ; causing a test runner to be deployed to the test instances ;
cause a test runner to be deployed to the test instances ; causing the software to be provisioned to the test
cause the software to be deployed to the test instances ; instances;
place the plurality of tests on a queue provided by a distributing the plurality of tests to the test runner
queue service executing in the service provider net deployed to the test instances wherein the test runner
work , wherein the test runner dequeues the plurality 60 performs the plurality of tests on the software in
of tests from the queue and performs the plurality of parallel on the test instances;
tests on the software in parallel on the test instances ; determining that the plurality of tests have completed ; and
determine that the plurality of tests have completed ; causing the test instances to be de -provisioned in response
and to determining that the plurality of tests have com
cause the test instances to be de -provisioned by the 65 pleted .
on - demand computing service in response to deter - 8 . The computer- implemented method of claim 7 ,
mining that the plurality of tests have completed. wherein the request to perform the plurality of tests com
US 9,811, 451 B1
27 28
prises an indication of testing preferences, and wherein the 16 . A computer - implemented method for performing a
number of test instances to utilize for performing the plu - plurality of tests of first software and a plurality of tests of
rality of tests on the software is determined based , at least in second software on a static fleet of test instances , the method
part, on the testing preferences. comprising:
9 . The computer-implemented method of claim 8 , 5 utilizing one or more heuristics to select a subset of the
wherein the testing preferences specify the number of test plurality of tests of the first software and a subsetof the
instances to utilize for performing the plurality of tests on plurality of tests of the second software ;
the software . making the selected subset of the plurality of tests of the
10 . The computer -implemented method of claim 8 , first software and the selected subset of the plurality of
wherein the testing preferences specify a maximum execu - 10 tests of the second software available to a test runner
tion time for the plurality of tests, and wherein the number executing on the static fleet of test instances , wherein
of test instances to utilize for performing the plurality of the static fleet of test instances include one or more
tests on the software is determined based , at least in part , on virtual machines or containers ;
historical test execution time data for the plurality of tests. causing the test runner to begin performing the plurality
11 . The computer- implemented method of claim 8 , 15
wherein the testing preferences specify a maximum execu of tests of the first software and the plurality of tests of
tion time for the plurality of tests, and wherein the number the second software in parallel on the static fleet of test
of test instances to utilize for performing the plurality of instances;
tests on the software is determined based , at least in part, on determining if the plurality of tests of the first software
historical test execution time data for tests similar to the 20 and the plurality of tests of the second software have
plurality of tests. completed ; and
12 . The computer- implemented method of claim 7 , in response to determining that the plurality of tests of the
wherein causing the determined number of test instances to first software or the plurality of tests of the second
be provisioned comprises instructing an on -demand com software have not completed ,
puting service executing in a service provider network to 25 utilizing the one or more heuristics to select a second
provision the determined number of test instances , and subset of the plurality of tests of the first software
wherein causing the test instances to be de-provisioned and a second subset of the plurality of tests of the
comprises instructing the on - demand computing service to second software , and
de - provision the test instances . making the selected second subset of the plurality of
13 . The computer - implemented method of claim 7 , 30 tests of the first software and the selected second
wherein distributing the plurality oftests available to the test subset of the plurality of tests of the second software
runner deployed to the test instances comprises placing the available to the test runner executing on the static
tests on a queue provided by a queue service executing in a fleet of test instances.
service provider network , and wherein the test runner is s 17 . The computer -implemented method of claim 16 ,
configured to dequeue a test from the queue and execute the 35
dequeued test on the software . wherein at least one of the one or more heuristics comprises
14 . The computer- implemented method of claim 7, fur a priority associated with the plurality of tests of the first
ther comprising: software and a priority associated with the plurality of tests
computing an estimated amountof time for completion of of1 the second software .
the plurality of tests ; 40 18 . The computer - implemented method of claim 16 ,
determining that the estimated amount time for comple wherein at least one of the one or more heuristics comprises
tion of the plurality of tests is less than a desired a desired time duration for completion of the plurality of
amount of time for completion of the plurality of tests ; tests of the first software and the plurality of tests of the
and second software.
causing one or more of the test instances to be de- 45 19 . The computer- implemented method of claim 16 ,
provisioned in response to determining that the esti wherein at least one of the one or more heuristics comprises
an estimated time for completion of the plurality of tests of
mated amountof time for completion of the plurality of the first software and the plurality of tests of the second
tests is less than the desired amount of time for comple software .
tion of the plurality of tests .
20
15 . The computer-implemented method of claim 7, fur- 50 wherein making . The computer- implemented method of claim 16 ,
ther comprising: the selected subset of the plurality of tests
computing an estimated amountof time for completion of of tests of the secondand software
of the first software the selected subset of the plurality
available to a test runner
the plurality of tests ; executing on the static fleet of test instances comprises :
determining that the estimated amount time for comple
tion of the plurality of tests is greater than a desiredod 5555 placing the selected subset of the
P first software and the selected subset plurality of tests of the
of the plurality of
amount of time for completion of the plurality of tests; tests of the second software on a queue provided by a
and queue service executing in a service provider network ,
causing one or more additional test instances to be pro and
visioned in response to determining that the estimated
amount of time for completion of the plurality of tests 60 wherein the test runner is configured to dequeue a test
is greater than the desired amount of time for comple from the queue and execute the dequeued test .
tion of the plurality of tests . * * * * *

Distributed Software Testing

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Distributed Software Testing

Uploaded by

Copyright:

Available Formats

MAIMUUNMUNTANT DE LA UNITA ATMATI HAITI

(12) United States Patent (10) Patent No.: US 9 ,811,451 B1

POLL QUEUE FOR A TEST 402

PERFORM THE TEST ON THE 410

GENERATE AND STORE TEST 412

126 1 2A 24A 112B 724B 1240126 1120 124D

DEVL RCOMPUTINGDEVICE SOFTWAREDVLOPER

SOFTWARE TESTING SERVICE

UTILIZE ON -DEMAND COMPUTING 206

DEPLOY TEST RUNNER 208

PLACE THE TESTS ON THE TEST QUEUE

UPDATE THE HISTORICAL TEST 214

DETERMINE NUMBER OF TEST INSTANCES

POLL QUEUE FOR A TEST

DEQUEUE TEST FROM 408

PERFORM THE TEST ON THE

GENERATE AND STORE TEST 4 412

DYNAMICALLY ADJUST NUMBER

COMPUTE AN ESTIMATE OF THE 504

112A1 2A 126 1 2B 16B 724B

108A DEVLOPERCMPUTINGDEVICE 108B DEVLOPRCMUTINGDEVICE 1080 DEVLOPRCMUTINGDEVICE

DISTRIBUTION OF TESTS FOR MULTIPLE CLIENTS

UTILIZE HEURISTICS TO SELECT A 704

PLACE THE SUBSET OF THE TESTS ONT 706

804A 804B 8040 804D

CDEANTEAR CDEANTEAR CDEANTEAR -

SERV COMPUTER NETWORKSERVICE SERVERCOMPUTER NETWORK SERVICE

POLICY SERVICE SOFTWAR STERVICNEG SQEURVICE

SCOMPUTING ERVICE STORAGE SRVICE SERVICE SERVICE

122 906F 906H 1024

1104 1108 1110

You might also like