2019 International Conference on Information Technology and Computer Application (ITCA)
Design and Research of Microservice Application
Automation Testing Framework
Yang WANG and Lei CHENG*
Information and Communications Testing and Simulation
Center
China Electric Power Research Institute
Beijing, China
hesoyam123456@sina.com
Abstract—Microservice style breaks through the traditional
monolithic application into a new generation of distributed
architecture, which realizes agile development and deployment.
How to ensure its quality and reliability by testing technology is
the focus in the software engineering. According to the
characteristics of microservice application, this paper analyzes
the test objects and test procedure, proposes an automation
testing framework based on modularization and data-driven, and
illustrates the main functions and processes of each layer.
Furthermore, a data structure based on the extended interface
description file is designed so as to generate test case data
automatically. This framework will contribute to establishing the
microservice testing platform and standard for SGCC (State
Grid Corporation of China), which is able to improve the
enterprise testing capability, raise work efficiency and achieve
effective management.
Keywords-microservices; automated testing; testing framework;
test data; test case generation
I. INTRODUCTION
With the rapid development of information technology, the
traditional software monolithic architecture has seriously
restricted the rapid update and iteration of projects.
Microservice architecture is a new generation of software
architecture which divides a traditional software system into
different modules. Each microservice focuses on a single
business and runs in an independent process. It uses lightweight
communication mechanism to cooperate with each other in
order to achieve a complete application. Microservices have
unique distributed architecture features, such as high cohesion,
low coupling, high fault tolerance and easy expansion, which
cannot fully meet the test requirements through traditional
testing methods [1]. In addition, microservices provide a kind
of machine to machine communication mechanism, so that it is
difficult to test them manually. Automated testing can replace
the traditional manual testing for functional and non-functional
testing, thus improving the testing efficiency greatly. Therefore,
in order to improve software quality, optimize work efficiency
and achieve effective management, it is of great theoretical and
practical significance to research an automation testing solution
and key technologies for microservice applications.
Supported by State Grid Corporation of China Science and Technology
Project (5700-201941223A-0-0-00).
978-1-7281-6494-6/19/$31.00 ©2019 IEEE
DOI 10.1109/ITCA49981.2019.00063
Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 18:28:52 UTC from IEEE Xplore. Restrictions apply.
Xin SUN
Power Technology Center
State Grid Zhejiang Electric Power Co., LTD. Research
Institute
Zhejiang, China
According to the test characteristics and test objects, a
hierarchical data-driven testing framework for microservice
applications is presented in this paper. Aiming at the test case
generation, an automatic generation method based on the
extended interface description file is designed. This research
will improve the software testing ability and accelerate the
implementation and promotion of microservice applications in
SGCC (State Grid Corporation of China).
II. MICROSERVICE APPLICATION TESTING ANALYSIS
A. Test Objects
Microservices have unique architectural characteristics such
as flexible technology selection, distributed structure, easy
expansion and controllable complexity. The internal design and
code implementation of the microservice application are
encapsulated and only the service interfaces are provided for
users. Therefore, the test is mainly based on the service
interface [2]. The test objects include the following two aspects.
1) Function Testing: Its main purpose is to test whether
the service is able to achieve the expected functional results
and whether the system can operate normally. The tester
compares the returned data with the expected result through
automatic recording, detection and playback so as to verify
whether the service can achieve the expected function.
2) Performance Testing: Its main purpose is to test the
service load by measuring the maximum number of requests
the service can handle, response time and so on. Microservices
divide a monolith into several services that run independently,
which increases the complexity of communication, so the load
of concurrent access by multiple clients is particularly
important.
B. Test Procedure
There are six processes in the general procedure of
automated testing for microservice applications: test
environment construction, software behavior modeling [3], test
case generation, test case execution, test result analysis and test
process improvement. In the testing framework, the modular
layered design idea can be used to separate the functions of
each test process, reduce the coupling of each function and
make it easy to manage and extend. Moreover, data-driven
257
technology can separate test logic from test data, remove the Data Layer
binding between data and scripts and improve the WADL
WADL File Parser Extended File
maintainability and reusability of the system [4].

C. Test Case Generation Input

Expected
Output
The key to improve the test automation lies in the test case Data
Data
generation automation. The service interface is described by
WADL (Web Application Description Language). Testers Data File WADL
Database
parse WADL files in order to get some information about the File
interface input parameters for automated testing. However,
Test Case Layer
only the parameter name and type are described in the WADL,
which lacks of test basis, so the WADL file needs to be Single Composite
extended. Reference [5], [6] and [7] put forward an automatic Parameter Test Parameter Test
test case generation method based on the extended interface Case Generation Case Generation
Microservice
description file, but they all lack of the complex constraint of a Application
parameter value. In reference [8], regular expression is Reverse Test Service
Function Case Filter Interface
proposed to describe parameter value constraints, but there is Test Script
no constraint between parameters. This paper will make some Generation Performance
improvements based on the previous studies. Test Script
Generation

III. AUTOMATION TESTING FRAMEWORK

Execution Layer Adaptation Layer
According to the characteristics, objects and processes of Function Performance Software
microservice testing, this section proposes a microservice Testing Tool Testing Tool Standard
application automation testing framework, as shown in Fig. 1.
The framework is based on the idea of layering and Hardware
Function Performance Standard
modularization. Each level is independent of each other and the Testing Testing
functions in every layer are modularized, which is good for
management and extension [9]. There are six layers in the Analysis Layer Management Layer
framework: adaptation layer, data layer, test case layer,
execution layer, analysis layer and management layer. The Test Result Collection Test Report Sending
framework is an overall solution for microservice testing. It
clarifies the task allocation, control process and dependency Function Performance Test Process
between components in the whole test, specifies the Verification Evaluation Measurement
architecture of the test program and provides support for
Configuration
component reuse. Test Report Generation Optimization

A. Adaptation Layer Figure 1. Microservice Application Automation Testing Framework

The adaptation layer establishes the interface standards of
software and hardware for data communication between the
microservice interface and the framework. The software
standard defines the format and communication protocol of
data sending and receiving. The hardware standard defines the
electrical index, time sequence and communication standard.
The mainstream microservices interact with each other in a
RESTful way, following the REST (Resource Representational
State Transfer) software architecture.
B. Data Layer
The data layer uses data-driven technology to organize the
input and expected output of the tested object in a certain form
to provide data for the test case layer. First of all, the WADL
extended file is resolved by the file parser. After that the test
data is generated for each input parameter according to the
principle of equivalence class division, random generation and
boundary value, and then the expected result data is generated
according to the validity of the test data. For performance
testing, sufficient additional valid data need to be generated.
All the test data can be saved in database and read through
different forms of data files.
The WADL file contains a small amount of information
about microservice interface parameters. In many cases,
parameters are not only self-constrained, but also mutually
constrained. Regular expression is a kind of logical formula
that can express rules flexibly and is very suitable for
expressing parameter constraints. The expansion of the WADL
file is accomplished by adding special XML (Extensible

258

Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 18:28:52 UTC from IEEE Xplore. Restrictions apply.
Markup Language) tags to the Decimal, String and Boolean
XML Schema basic data type parameters and using regular
expressions to describe parameter constraints. The structure of
each parameter is shown in Fig. 2. The name, type and required
represent the name, type and nullability of a parameter
respectively. These are the original parameter information in
the WADL. The expression depicts the regular expression for
parameter value and constraints means the constraint queue of
the current parameter and other parameters. Each constraint
constrain is a triplet, including the current parameter value, the
constrained parameter name and the regular expression of the
constrained parameter value.
name
type value
constrain
required constrain
constraint c_name
expression c_expression
constraints
Figure 2. Parameter structure in WADL file
C. Test Case Layer
The test case layer automatically generates test cases using
data read from the data layer and then generates test scripts for
the executive layer. The test script is a kind of executable data
file that can be generated automatically by test tools or
programmed manually. The test script can be divided into
functional script and performance script for different test
objects. Before the performance script is generated, the reverse
test cases need to be filtered out through the validity of the
parameters and only valid test cases are reserved. It is noted
that the script itself does not contain test data, which is easy to
maintain and extend.
The automatic test case generation method varies according
to the parameter type. There are two types of parameters about
microservice application interface: single parameter and
composite parameter. For the single parameter, test cases are
generated automatically one by one for each values of the
parameter. Composite parameter means that any one parameter
must appear in the specified order and only once, therefore the
various combinations of parameters need to be covered in
generating test cases.
If there are k parameters in the service interface and each
parameter has n values, then the number of fully combined test
cases is nk which could be so huge. The pairwise method can
transform the parameter complete combination into the
parameter pair coverage, which is able to reduce the number of
test cases greatly and is considered an effective test method
[10]. The following definitions are made in this paper. For a
microservice application interface, its parameters can be
expressed as a set P, where the value of each parameter pi can
be represented as a set V. For a value vj of a parameter pi, its
structure can be described as a quadruple, namely,
The name indicates the parameter name and the value
depicts the parameter value. The isvalid is a boolean type,
indicating whether it is a valid parameter value. The
constraint_list is a set that represents a list of constraints on
other parameters when the parameter is in the current value.
Each constraint in the constraint_list includes two parts of the
constrained parameter: its name and the regular expression.
Typical pairwise testing algorithms include AETG and IPO
[11]. Both of them need to generate uncovered pairing set UC
and then generate test cases iteratively through different
strategies until all pairs of UC are covered. Based on the
parameter structure and constraint between parameters
designed in this paper, the following two combination
principles are formulated:
x By obtaining the isvalid of each parameter, it is
ensured that only at most one parameter in a parameter
combination is invalid.
x When combining parameter pm and pn, if the
constraint_list of pm is not empty and it contains pn, It
should be ensure that there is no combination of pm and
pn other than the constraint list.
In this way, the combination of invalid parameters can be
avoided and also the number of UC can be reduced, thus the
number of test cases will be reduced so as to effectively avoid
the explosion of data cases caused by combination of
parameters.
D. Execution Layer
In the execution layer, the test script is imported into the
test tool so as to perform automated test execution.
Microservice application interface automation testing tools
include WebInject, WStest, SoapUI, TestMaker, QEngine,
JMeter and so on [3][12]. During the function script execution,
the scripts are executed one by one according to the execution
strategy preset by the tester and an exception will be thrown
when a test case does not meet the expected result. Before the
performance script execution, the tester sets the threshold value
for the utilization rate of hardware resources, response time and
other performance indicators in order to use them evaluate the
test results.
E. Analysis Layer
The analysis layer includes some analysis operations after
executing test scripts. Test results should be collected firstly,
including the description of each test case and its execution
return data. After that, the function can be verified. Testers
check the test execution process that does not meet the
expected results and then analyze the specific defect reasons.
On the basis of the preset threshold, the performance test can
be evaluated and the cases that do not meet the performance
standards can be analyzed in depth. Finally, the test report is
generated.
F. Management Layer
In the management layer, test reports will be distributed and
pi.vj = (name, value, isvalid, constrain_list).   sent to the test related stakeholders, such as project manager,
test director, development engineer, etc. After the entire test

259

Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 18:28:52 UTC from IEEE Xplore. Restrictions apply.
framework has been running for a period of time, some
quantifiable attributes of the test process can be collected and
analyzed so as to measure the testing process. Finally, the test
process of the microservice applications can be improved by
optimizing the test configuration.
IV. CONCLUSION
In this paper, a microservice application automation testing
solution is proposed and a testing framework is designed based
on modularization and data-driven technology. The framework
completes the automated testing of the microservice interface
through coordination among the adaptation layer, data layer,
test case layer, execution layer, analysis layer and management
layer. The extended WADL file structure and parameter data
structure are designed for the test case data generation, which is
the key part of automation test, and also the parameter
combination coverage problem is optimized. In the future, a set
of microservice application test tools can be developed based
on this framework in combination with relevant standards of
SGCC, which contributes to forming a unified test standard and
platform for the enterprises.
ACKNOWLEDGMENT
This research was supported by State Grid Corporation of
China Science and Technology Project (5700-201941223A-0-
0-00). We would like to acknowledge our sponsors for their
financial support.
260
Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 18:28:52 UTC from IEEE Xplore. Restrictions apply.
REFERENCES
[1] X. W. Wang and L. Du, “State Grid Corporation of China Science and
Technology Project Feasibility Study Report on Testing System and Key
Technologies for Microservice Applications,” unpublished.
[2] X. G. Feng and J. X. Liu, Review on Web Service Testing Technologies,
Microcomputer Applications, vol. 31, no. 1, pp. 21-26, Jan 2010.
[3] C. Y. Ma, Y. A. Zhu and W. Lu, Automatic Test for Web Services,
Computer Science, vol. 39, no. 2, pp. 162-169, Feb 2012.
[4] S. G. Li and X. M. Ding, A Method of Automatic Test Data Generation
Based on Extended WSDL, Journal of Southwest China Normal
University(Natural Science Edition), vol. 36, no. 1, pp. 188-192, Feb
2011.
[5] X. L. Yuan, “Automatic Test Cases Generation Based on Extended
WSDL,” Diss. Southwest University, May 2009.
[6] N. Q. Jiang, “Automatic Test Case Generation Method Based on
Extended WSDL for Web Services,” Diss. Hefei University of
Technology, Apr 2012.
[7] S. G. Li, “A Method of Automatic Test Data Generation Based on
Extended WSDL,” Diss. Southwest University, May 2010.
[8] Z. H. Hao, “The Research of Auto Web Service Test Based on Regular
Expression Extension,” Diss. South China University of Technology,
May 2012.
[9] M. Dong, “Research on the Technology of Test Automation Scripts
Genertion for Web Services,” Diss. Shijiazhuang Tiedao University, Jan
2014.
[10] Y. J. Wu, “Analyses and Implementing of Algorithms for Pairwise
Testing,” Diss. Huazhong University of Science & Technology, Jan
2008.
[11] F. Liang and X. Q. Song, Method of Generation Pair-cover Test Cases
Based on AETG Algorithm, Computer Engineering and Design, vol. 35,
no. 11, pp. 3850-3854, Nov 2014.
[12] P. F. Liu, “Design and Implementation of Web Service Testing Tool,”
Diss. Inner Mongolia University, Jun 2016.