You are on page 1of 4

ClearTH Test Automation Framework for

DLT-Based Post-Trade Systems

Vladimir Panarin∗ , Alyona Bulda∗ , Iosif Itkin∗ , Murad Mamedov∗ , Alyona Rybakova∗ , Anna Gromova∗ ,
Alexey Zverev∗ , Elena Treshcheva∗ , Sergey Tishin∗ , Kirill Zagorouiko∗ , Rostislav Yavorski∗†
∗ Exactpro Systems, London/Moscow/New York, UK/Russia/USA
† Higher School of Economics, Moscow, Russia

Abstract—The paper presents an overview of a test automation The challenges outlined above and the complexity of mi-
framework aimed at end-to-end functional and non-functional grating financial business as usual (BAU) processes to new
testing of DLT-based hybrid financial software for post-trade. The software platforms necessitate their professional testing, chal-
proposed solution comprises the components designed for testing
user-facing parts of the SUT as well as business logic specific lenged itself by the diversity of blockchain-inspired solutions
for different DLT-based architectures. This combined approach across the industry [9]. An exhaustive yet universal approach
is seen as a viable solution of the problem of the SUT complexity proposed in this paper takes into account the challenges
as well the variety of possible DLT architectural decisions. posed by existing blockchain solutions while covering the use
Index Terms—distributed ledger technology, blockchain, soft- cases common for post-trade business practices. Given the
ware testing, quality assurance, test automation
complexity and architectural variety of DLT-based financial
infrastructures, the solution is proposed in the form of a
multifunctional test framework rather than a single blockchain-
Nowadays, blockchain and distributed ledger technologies oriented testing tool.
(DLT) leverage solutions for almost every business area from The paper is organized as follows:
supply chains to government and finance. As outlined in - Section II contains an overview of related work in form
research papers and business white papers, the technology of both research papers and existing tools;
underpinning blockchain promises a lot of benefits through - Section III introduces an example of a simplified DLT-
secure, tamper-proof, peer-to-peer decentralized networks with based post-trade system (A) and a description of a business
distributed consensus [1]. Along with removal of intermedi- flow (B) as well as proposes the ClearTH test automation
aries, flat data structures that will reduce the lags of reconcili- framework and End-to-End test approach (C);
ations among different local ledgers, compressed confirmation - Section IV conveys a conclusion.
times and near real-time settlement of transactions, blockchain
as a technical solution is seen as a means of providing data II. R ELATED W ORK
and transaction immutability, resiliency against cyber-attacks The papers addressing DLT outline advantages as well as
and fault tolerance [2]. The trends of decreasing inefficiencies challenges posed by the new technology. Among these publi-
and costs of trading and reconciliation processes along with cations there are both technical papers discussing diversity and
increasing transparency between the market participants seem limitations of blockchain solutions and business white papers
to be the ones that revolutionize the financial industry [3]. which give an overview of approaches already used in the
However, there is a number of challenges that the financial financial industry and/or provide examples of use cases of DLT
institutions face when they introduce blockchain components applications in business practice.
into existing business operations [4]. The technical papers mention such limitations of
Firstly, there are challenges in the post-trade infrastructure blockchain-based solutions as not 100-percent guarantee
itself [5], such as a large number of components, upstream and of transactions immutability, issues with blockchain networks
downstream system dependencies, complexities of participant throughput, latency, size and bandwidth, security, wasted
structure, trades/positions/accounts lifecycle, asset classes, risk resources, usability, versioning/hard forks/multiple chains,
calculation and access via a variety of API endpoints. trustworthiness, etc. [10], [11].
Secondly, there are challenges caused by adoption of DLT The business papers outline that the complexity and lack of
into post-trade platforms [6], [7] combined with business experience associated with blockchain-based systems are the
model changes, new demands for segregated service delivery, obstacles to their wide adoption. Additionally, significant work
client-driven flexibility, performance, SLAs and new develop- and investment are required for financial services organisations
ment practices (agile, CI/CD) [8]. to be technically prepared to make DLT solutions part of
Lastly, blockchain technologies, being innovative solutions, platforms which still have legacy components and ensure their
pose limitations due to the lack of research and practices smooth integration [12].
focusing on software quality assurance and test automation As for the research papers with focus on testing of
of DLT-based post-trade platforms. blockchain, we have reviewed a library of recent research in
this knowledge area. ACM digital library ( III. RUNNING E XAMPLE
and IEEE Xplore ( A. Design of the DLT-based application
were used as a source for the search with a criterion for the
papers to contain both the words blockchain and testing. The In order to develop an End-to-End approach and prove that
retrieved set was filtered so that it contained only the works ClearTH Test Automation framework fits the purpose of the
focused on description of the test method, test tools, quality proper quality coverage, a simplified DLT-based application
characteristics or the test results, which yielded 25 research has been designed. The post-trade area has been chosen for
papers published in 2017-2018. Such a number serves as an this running example, because nowadays there are still a lot
indication of the relevance of the task at hand and reflects the of challenges around process automation, systems capacity,
fact that research in this area is still at the inception phase. resilience, maintenance and many other aspects. On the other
hand, there are plenty of the technology types created for solv-
The papers can be split into three categories: 1) dedicated to
ing these challenges and on-boarding as many manual routine
functional testing - 24%, 2) dedicated to non-functional testing
activities as possible. One of such open source blockchain-
and benchmarking - 36%, and 3) covering both areas - 40%.
inspired platforms, namely R3 Corda, has been selected for
As it is shown, to ensure that a blockchain-based system is
the purpose of implementation of this running instance. Its
trustworthy two types of test suites are needed: covering smart
architecture is illustrated by the Fig. 1.
contract testing and blockchain transaction testing [13].
One of the approaches to testing of smart contracts sug-
gests adopting TDD (Test-Driven Development) or BDD
(Behaviour-Driven Development) practices to the smart con-
tract development process [14].
Overall, the approaches proposed in the papers do not solve
the problem of complexity and diversity of parameters in
different blockchain-inspired solutions.
As far as existing tool implementations are concerned,
among them there is a number of those that cover the func-
tional aspect of use cases. Most of them are focused on smart
contract testing: automated contract testing with monitoring of
the execution of smart contracts and information extraction for
vulnerability analysis [15], detection of re-entrancy defects in
smart contracts [16], and others.
There are also blockchain-oriented tools that cover non-
functional aspect of testing: among the problems addressed by Fig. 1. Test approach tailored for Corda.
these tools there are security and vulnerability [17], high per- The test automation approaches covered by this architecture
formance consensus mechanism [18], measuring blockchain include two types of the test methods: a transactional test
performance and security [19], test environment manage- approach with opportunities for emulating the surrounding
ment and test repeatability [20], overall property-testing for (integration) systems and post-transactional one. As a rule,
blockchain solutions [21] and gateway testing of two different post-transactional test automation methods are used as an
blockchain ecosystems through containerization [22]. added value to the main test automation (functional and
Thus, even though there is a massive variety of tools that can non-functional) frameworks in order to provide analysis of
be used to ensure high standards of quality and mechanisms captured data and perform its integrity checks. Going further,
of testing which are largely similar, none of the tools outlined the transactional test automation framework will be detailed in
above focus on end-to-end functional testing along with a this paper as a first step to building a regression library with
regression library for a post-trade infrastructure based on DLT. sufficient coverage which is suitable for parallel execution of
Unlike the other tools, the testing solution described in complex multi-day scenarios.
this paper represents a test automation framework capable of
testing post-trade systems of any level of complexity. While B. Design of business flow case for the DLT-based post-trade
being focused on testing hybrid systems that comprise dApps application
communicating with the users and the external world, as well The business flow case selected for the running example
as blockchain-specific components of a closed system (smart purposes is designed in a simplified form. However, it is scal-
contracts), the solution provides an extensive regression testing able in terms of complexity: multi-days cycles, dependencies
library. Along with its satellite for non-functional testing, the on the other system processes and their states as well as the
framework not only covers non-functional aspects of post- state of the DLT-based application can be added at any time.
trade use cases, but also provides end-to-end functional testing The life cycle of the designed business flow supports two
based on business requirements for post-trade systems built on operational models: 1) when there is no flow initiating trigger
DLT [23]. in the system and all the participants have to initiate their
involvement themselves, and 2) when there are initial pre- possible. The analysis and test execution from the end-to-end
conditions provided in order to trigger application functioning perspective is important and provides the overall picture of the
from the onboarding point. The second model is usually system’s under test quality and readiness for the real-world
required when the business service is migrating from one conditions which are likely to be faced in production. Hence
system to a newly developed cutting-edge technology system. a question: how to test the business flow scenario described
An ability to operate in both models allows our test library be above from the end-to-end perspective.
re-executable at any moment without additional preconditions Let’s imagine that there is no possibility to have Bank B
in place for this running example. from the Fig. 3 in place for the integration testing. In the
position update scenario, the absence of the participant-node
practically blocks the whole flow. And one additional end-
point of the designed DLT-based system for the testing is
market maker, whom the test automation framework should
be able to emulate in order to trigger the business flow.
The ClearTH test automation framework allows to emulate
the missing end-points and therefore helps to continue to vali-
date the next steps of the business flow. The stubs incorporated
into ClearTH have no dependencies on whether the system
under test is DLT-based or not, and therefore helps to mitigate
the difficulties arising from the integration with ETL layer
(Extract, Transform, Load).

Fig. 2. Business flow for the DLT-based post-trade system.

The business flow case illustrated on the Fig. 2 represents a
blockchain network which consists of participants, i.e. nodes.
Each node is governed by the algorithms responsible for
different parts of the running example flow.
The starting point of the selected scenario is the market op-
erator entering the trades which, in their turn, trigger consensus
algorithms on the shared ledger. There are three members of
the blockchain network: two banks with participant roles and
a Central Counterparty (CCP), and 2 types of the statements
which should be updated by the ledger: trade and position.
For the position update flow in the demonstrated example,
a relevant logic was developed to update position states in
accordance with the received details of the trade between the
banks: asset class, currency rate, timestamp, quantity, price, Fig. 3. ClearTH Test Automation Framework and test ap-
sides, participants, etc. The algorithm reaches the completion proach for the DLT case study.
only after receiving signs from the each participant involved in When end-to-end approach is selected for the test library
accordance with the distributed ledger technology approach. and automation framework, their structure considers the sys-
This business flow is prepared for adopting the ClearTH tem life cycle from the beginning. The related schedule
Test Automation Framework designed inline with the end-to- mechanism contains main configurations of the test automation
end test approach. framework and the system under the test. This approach allows
multiple users to test the application simultaneously. Each
C. ClearTH Test Automation Framework and End-To-End test scenario represents the business flow with its own schedule
approach for the DLT-based post-trade application dependencies and can be incorporated into the common design
A proper approach should be able not only to secure the schedule of the ClearTH test automation framework. The
project with a good test library, but also to support quick scenario itself consists of the global steps outlining dependen-
change requests to the application under test and easy mainte- cies on the schedule and steps governing validations, event
nance of the test library. Such a framework should cover a lot triggering and emulation. The emulation stubs mentioned
of aspects of the definition of the software quality ranging from above are incorporated into each node as a ClearTH satellite,
the end-to-end coverage to ability to execute test scenarios in Woodpecker. Such an architecture of the framework allows its
different system states in order to detect problems as soon as usage per both functional and non-functional test approaches.
In the running example, there are eventually three global [3] Biondi D., Hetterscheidt T., Obermeier B. Blockchain in the Financial
steps: 1) start of the flow, 2) trade generation, and 3) positions Services Industry. 2017.
[4] Radziwill N., Benton M. Quality and Innovation with Blockchain
update. Every state change after each of the global steps Technology. In: Software Quality Professional Magazine. 2017. V. 20.
is covered by a set of validations triggered by running the No.1.
scenario: the position state of Bank A in the vault of Bank A, [5] Rewiring Securities Post-Trade Challenges and Oppor-
tunities in the New Order, 2017. [Online]. Available:
the position states of both banks on CCP side, the state of the
trade shared on the ledger, etc. This is done to narrow down challenges-and-opportunities-new-order. [Accessed: 23-Jan-2019].
the system deviation and localise the defect. [6] Mills D., Wang K., Malone B., Ravi A., Marquardt J., Chen C., Badev
A., Brezinski T., Fahy L., Liao K., Kargenian V., Ellithorpe M., Ng
The Fig. 3 schematically shows the ledger-related states and W., and Baird M. Distributed ledger technology in payments, clearing,
algorithms of the DLT-based post-trade system under test as and settlement, Finance and Economics Discussion Series 2016-095.
well as validation and emulation end-points of the ClearTH Washington: Board of Governors of the Federal Reserve System.
[7] Santo A. et al. Applicability of distributed ledger technology to capital
test automation framework. market infrastructure // Japan Exchange Group, Working Paper. Oct.
For the purposes of an exhaustive test coverage, the schedule 2016. .3.
also allows to have scenarios be multiplied at each global [8] Marchesi M., Marchesi L., and Tonelli R. 2018. An Agile Software En-
gineering Method to Design Blockchain Applications. In Proceedings of
step by the variety of the events and responses. Having in the 14th Central and Eastern European Software Engineering Conference
place schedule structure, stubs and ability to execute multiple Russia (CEE-SECR ’18). ACM, New York, NY, USA, Article 3.
scenarios in one go, the ClearTH test automation framework [9] X. Xu et al., “A Taxonomy of Blockchain-Based Systems for Ar-
chitecture Design,” 1st IEEE International Conference on Software
is capable to monitor and control the nodes of the DLT- Architecture (ICSA 2017), Gothenburg, Sweden, 2017. Pp. 243-252.
based post-trade systems, re-execute the regression library [10] Weber I., Gramoli V., Ponomarev A., Staples M., Holz R., Tran A.
and validate the system capacity, resilience and other non- B., Rimba P., On Availability for Blockchain-Based System, IEEE 36th
Symposium on Reliable Distributed Systems, Sydney, Australia, 2017.
functional parameters of the platform. Pp. 64-73.
In addition to the described capabilities, the test automation [11] Swan M. Blockchain: Blueprint for a New Economy. OReilly Media,
framework is able to provide test reports, test execution statis- Inc. 2015. 129 p
[12] Swift. gpi Real-Time Nostro. Proof of Concept. [Online]. Available:
tics, system monitoring states and non-functional measurement report nostro
characteristics which help in further analysis. public release .pdf. [Accessed: 23-Jan-2019].
[13] Simone Porru, Andrea Pinna, Michele Marchesi, Roberto Tonelli,
IV. C ONCLUSION Blockchain-oriented Software Engineering: Challenges and New Di-
rections, IEEE/ACM 39th IEEE International Conference on Software
The ClearTH test automation framework was developed Engineering Companion, 2017. Pp. 169-171.
in 2011 and used to test clearing and settlement platforms. [14] Chun-Feng Liao, Ching-Ju Cheng, Kung Chen, Chen-Ho Lai, Tien Chiu,
and Chi Wu-Lee, Toward A Service Platform for Developing Smart Con-
The first Proof of Concept for the DLT-based systems was tracts on Blockchain in BDD and TDD styles, IEEE 10th International
conducted in October 2018. It addressed a use case of trade Conference on Service-Oriented Computing and Applications, Taipei,
sharing between the participants and a CCP and a relevant Taiwan, 2017. Pp. 133-140.
[15] Bo Jiang, Ye Liu, W.K. Chan, Contract Fuzzer: Fuzzing Smart Contracts
position update algorithm within the blockchain network. for Vulnerability Detection, Beijing, China and Hong Kong, 2018, Pp.
The proposed test framework is able to adapt to challenges 259-269.
presented by DLT-inspired architectures and test post-trade [16] Chao Liu, Han Liu, Zhao Cao, Zhong Chen, Bangdao Chen, and Bill
Roscoe. 2018. ReGuard: Finding Reentrancy Bugs in Smart Contracts.
systems of any level of complexity. While being focused on In ICSE 18 Companion: 40th International Conference on Software
testing hybrid systems that comprise dApps communicating Engineering, May 27-June 3, 2018, Gothenburg, Sweden. ACM, New
with the users and the external world as well as blockchain- York, NY, USA, 4 pages.
[17] Reza M. Parizi, Ali Dehghantanha, Kim-Kwang Raymond Choo, Amri-
specific components of a closed system, it provides an exten- traj Singh. 2018. Empirical Vulnerability Analysis of Automated Smart
sive regression testing library which orchestrates functional Contracts Security Testing on Blockchains. In Proceedings of 28th
and non-functional approaches for DLT-based post-trade ap- Annual International Conference on Computer Science and Software
Engineering, Canada, October 2018 (CASCON18), 11pages.
plications. [18] Nida Khan, FAST: A Map Reduce Consensus for High Performance
The objective of the ClearTH test automation framework is Blockchains, In The 1st Workshop on Blockchain-enabled Networked
to become an open source reference platform for testing DLT- Sensor Systems (BlockSys18), November 4, 2018, Shenzhen, China.
ACM, New York, NY, USA, 6pages.
based applications across multiple fabrics available. Currently, [19] Jason Spasovski, Peter Eklund, Proof of Stake Blockchain: Performance
it is being enhanced, and the team is planning to contribute and Scalability for Groupware Communications, Copenhagen, Denmark.
the core part of it as well as the DLT related plug-in into open Pp.251-258.
[20] Walker M. A., Dubey A., Laszka A., Schmidt D. C., PlaTIBART: a
source later in 2019. Platform for Transactive IoT Blockchain Applications with Repeatable
Testing, M4IoT17, December, 2017, Las Vegas, NV, USA. Pp.17-22.
R EFERENCES [21] Chepurnoy A., Rathee M., Checking Laws of the Blockchain with
[1] Koul, R. Blockchain Oriented Software Testing. In: Challenges and Ap- Property-Based Testing, IWBOSE, Campobasso, Italy, 2018. Pp. 40-47.
proaches, 3rd International Conference for Convergence in Technology [22] Galkin, R., Staroletov, S. (2018). Cryptocurrency Gateway Testing
(I2CT), Pune, India, 2018. Pp. 1-6. Technology. [Online]. Available: /publi-
[2] Biella M., Zinetti V. (UniCredit). Blockchain Technology cation/330009756 Cryptocurrency Gateway Testing Technology. [Ac-
and Applications from a Financial Perspective. Technical cessed: 31-Jan-2019].
Report. Version 1.0. February 26, 2016. [Online] Available: [23] ClearTH Test Automation Framework for DLT-Based Post-Trade Sys- tems, 2019. [Online]. Available: [Ac-
Applications-from-a-Financial-Perspective. [Accessed: 23-Jan-2019]. cessed: 07-Jan-2019].