White Paper

Approach to Testing SOA

Applications
Last Updated: 7th May, 2007
Service Oriented Architecture
Introduction
Service Oriented Architecture (SOA) makes software quality
both more important and more difficult to achieve. But
traditional approaches to software testing are insufficient
in an SOA environment. IT organisations pursuing SOA
find that they must rethink their testing methods and revise
testing roles and responsibilities.
This White Paper aims to identify the risks and challenges
organisations face and highlights an approach to taking
these on.
What is SOA?
SOA is “an architectural style whose goal is to achieve loose
coupling among interacting software agents/services ”. A
service is a unit of work performed by a service provider
to achieve desired end results for a service consumer.
Both provider and consumer are roles layered by software
agents on behalf of their owners. SOA is a new paradigm
that supports modularised implementation of services
(business logic). This architecture is particularly applicable
when multiple applications running on varied technologies
and platforms have to communicate with each other. The
figure below shows the basic components of SOA:
The components of SOA include:
Service providers: A service provider is a component or
set of components that execute a business function in
a stateless fashion, accepting predefined inputs and
outputs.
Service consumers: A service consumer is a set of
components interested in using one or more of the services
provided by service providers.
Service repository: A service repository contains the
descriptions of the services. Service providers register their
services in this repository and service consumers access
the repository to discover the services being provided.
AppLabs.com
App_WhitePaper_Approach_to_SOA_1v04 Page 
Web Services, Business Process Expression Language
(BPEL) and Enterprise Service Bus (ESB) enable the
implementation of SOA in a novel way. Emergence of the
web and the responsiveness of the business applications to
continuously changing business situations have given birth
to event oriented thinking while designing applications. The
events are to be defined as hardware, operating system,
network, application and interfaces to other applications.
The events trigger a set of actions or utilise the notification
service within the SOA. Event Driven Architecture (EDA) is
still in its infancy.
SOA Applications
Traditional application centric design focuses on user
interface, process logic and data. A robust architecture that
supports events and designs business logic as services
in a heterogeneous application landscape introduces a
business process layer to manage the process steps and
integration layer to interoperate the applications providing
these services. The diagram below illustrates the five tier
architecture propounded by Gartner.
New applications are being built using the five tier
architecture to enable them with SOA and EDA.
Composite Applications
Enterprises have an inventory of applications (SOA
applications and non-SOA applications) to support their
business processes. Extending the business processes to
collaborate with their value chain partners (suppliers and
customers) and other stakeholders (investors, employees
and regulatory bodies) to achieve real time agility and
regulatory compliance is resulting in an environment
© 2007 AppLabs
of composite applications within an enterprise. Testing
SOA applications encompasses testing the functionality,
interoperability, security, performance and other non-
functional aspects of the composite applications.
Benefits, Challenges and Risks
Business Benefits
Services as an architectural thinking has been around for
a while, with the introduction of CORBA???, COM/DCOM,
and RMI. Due to the inherent limitations of the technology
with respect to cross platform integration, the concepts
were not widely adopted. With the evolution of internet
and its associated technologies (XML, SOAP, UDDI etc) the
implementation of SOA (methodology) has gained impetus
with web services (execution strategy). Major business
objectives envisioned with SOA applications are as follows:
Ñ Reuse IT assets;
Ñ Integrate heterogeneous systems;
Ñ Real time agility;
Ñ Collaboration.
Challenges
The new paradigm adopted by the development community
has introduced more challenges in terms of testing the
software built under the SOA philosophy. Following are
some of the top challenges testing SOA applications:
Ñ Real time applications;
Ñ No user interface for the services;
Ñ Conceptualizing test bed close to the end user
environment;
Ñ Infinite consumers possible (thick client, thin client
etc.);
Ñ Infinite user population;
Ñ Multi-skilled test teams: domain, technology and testing
knowledge.
Risks
Following are the top risks testing SOA applications:
Ñ Functionality;
Ñ Performance;
AppLabs.com
App_WhitePaper_Approach_to_SOA_1v04 Page 
Ñ Backward compatibility;
Ñ Interoperability;
Ñ Compliance;
Ñ Security.
Understanding business objectives, testing challenges
and analysis of testing risks is important to determine an
adequate and accurate test approach.
Testing Approach
A successful test approach is one which ensures the
business benefits of the software solution are achieved.
This section highlights the testing challenges faced in the
following two scenarios:
Ñ Existing applications are modified (service enabled)
so that application to application communication is
made possible; ISVs develop and release service packs
or feature packs to make their applications service
oriented. Enterprises research opportunities to address
the disparities in their current applications;
Ñ New applications will be “designed” ground up with
service orientation as a goal rather than as an after
thought.
Both the scenarios have similar concerns but differ when it
comes to the amount of testing effort and test techniques
employed.
Functionality
Service oriented applications are built with interoperability
in mind, resulting in business processes that span across
departments (intra) and organisations (inter). The test designs
have to be specified in what we call a “funnel approach”,
ensuring the functional tests address end-to-end business
flows. The specified flows are dissected to isolate the inter-
application tests. From the inter-application tests, intra-
application tests are discovered, finally resulting in unit tests.
Test execution is driven in the traditional fashion from unit
to end-to-end tests; this ensures that every unit is delivered
with its desired functionality. Though unit tests are a must,
the most critical are the integration tests. Integration tests
must be considered at two levels:
Ñ Inter-application;
Ñ Intra-application.
© 2007 AppLabs

Let us take an example to illustrate the funnel approach in
the perspective of an extended enterprise which helps to
integrate customers and suppliers. This typically involves
the following steps:
Ñ Customer uses the order enquiry web service of the
enterprise sales application with an enquiry of his
requirements;
Ñ Order enquiry web service of sales application checks
with the Available to Promise (ATP) web service of order
fulfillment application;
ü If there is enough material in stock that meets the
delivery requirements of the customer, enterprise’s
ability to fulfill the order is communicated back;
ü If there is not enough material in stock that meets
the delivery requirements of the customer, Capable
to Promise (CTP) web service of the order fulfillment
application is triggered;
Ñ CTP web service in turn triggers capacity CTP web
service from the order fulfillment application and an
external component CTP web service from supplier
application;
Ñ Supplier confirms back the commitment dates for the
quantities mentioned;
These steps illustrate the order promising business flow.
To utilize the funnel approach, business experts create test
scenarios which would encompass the end to end path. The
required test data and expected results are also documented
at each step. Once this step is accomplished the test
scenarios are functionally decomposed to understand the
inter application entities, the next step is to further break
down the inter application entities to intra application
entities finally boiling down to the units. In conjunction with
the funnel technique, AppLabs recommends exploratory
testing techniques (by bringing business experts) as they
are very effective instruments to unearth defects which
cannot be detected by other techniques.
Most of the commercial SOA test tools focus heavily on
unit testing. Some of the vendors offer business process
testing using the same unit tests (basically weaving
together the units to form intra application bound tests).
Generally, integration tests could be carried out using the
normal functional tests if they possess the same or a similar
technology stack.
Performance
The greatest challenge when it comes to performance
testing and service oriented applications is to design the
end user workload and model the end user environment
(applications involved, software, hardware etc). The
predominant practice is to define and design usage modeling
for the entire suite of applications utilising Markov chains.

Ñ Capacity CTP web service confirms back the

commitment dates for the capacities mentioned;

Ñ Consolidated CTP information is sent back to the order

enquiry web service via the ATP web service;

Ñ Once the customer is happy with the returned ATP/CTP

information, then the customer can confirm the order.
Customer invokes the order entry web service of the
sales application;

Ñ Sales application completes the order entry.

AppLabs.com
App_WhitePaper_Approach_to_SOA_1v04 Page  © 2007 AppLabs
As shown in the picture below, stake holders and product
managers are quizzed on the usage of the application. This
would be used as a subjective assessment to support the
analytical approach of Markov chains. The data of usage
probability is unearthed from raw data in the form of web
server logs and application server logs. This will help to
design a realistic model that closely replicates end user
usage.
Once the appropriate performance scenarios have been
defined, multiple test tools/techniques are required because
of the presence of different platforms and technologies.
During test execution, monitoring application performance
and collating data would be a challenge since there is no
“one stop shop” tool which gives insight into the overall
big picture.
For applications which have been service enabled as an
afterthought, the obvious challenge is to retain the same
quality of service measures and service level agreements
when compared to the previous applications (since new
layers have been enabled, deterioration of performance is
predicted).
For applications built with service orientation from the
ground up, performance tests have to be carried out at
the unit level.
Backward Compatibility
Services operate under contract (an interface enables
functionality to publish the contract); and service
consumption is based on this contract. The advantage of
SOA is the ease of changing the implementation without
breaking the contract. As businesses evolve, software
infrastructure needs to adapt to the change. Some of
these changes may require that the contract be broken.
The challenge of infinite service consumers poses a serious
AppLabs.com
App_WhitePaper_Approach_to_SOA_1v04 Page 
challenge to test and make sure that old service consumers
are still supported. A solution for this challenge is to possess
test assets which are automated, representing the universe
of consumers who are expected to be supported. These
test assets would be run against the new service to ensure
backward compatibility.
Interoperability
SOA became successful when the integration infrastructure
started adopting standards such as SOAP, XML etc. This
helped different platforms in an enterprise to talk to each
other (e.g. J2EE and .NET; different operating systems).
Though this is a boon for the CIOs, it is a challenge on the
application development groups to ensure interoperability.
In application to application communication, data transfer
is the key and data is stored, transmitted and manipulated
in a proprietary method (e.g. Big Decimal data type in
MS and Apache World are not compatible, the SOAP
headers are not identical on all the platforms). To simulate
interoperability tests, building a test environment which is
close to a real life scenario is a great challenge. Taking
advantage of software virtualization is one good approach.
Crafting reusable test assets that work on many different
platform combinations would also be of added value.
Compliance
SOAs (especially web services) are currently using specific
standards to ensure that they are interoperable. Due
to the proliferation of productivity tools (development
toolkits); there is room for heterogeneity being added to
the integration infrastructure. This, however, might lead to
non compliance of the various standards adopted in the
context of W3C, WS-I, XML, and SOAP etc. As soon as the
components are available, compliance tools can be run to
ensure compliance with the different standards.
Security
SOAs implemented via web services have very similar
challenges to those which web applications face (for
example, cross site scripting, buffer overflows, SQL injection
etc). The greatest security challenge is in web services
which are open to public access where the universe of
users is unlimited (contrary to the enterprise scenario
where users are limited and can be authenticated).
Development phases
The following table shows the development phases and
the applicable test types in each phase:
© 2007 AppLabs
 Backward Inter-
Phase Functionality Performance Compliance Security
Compatibility operability
Inception
Review Requirements
¸ ¸ ¸ ¸ ¸ ¸

Elaboration
Functional/ Technical
¸ ¸ ¸ ¸ ¸ ¸
Design
Construction
Review Code ¸ ¸ ¸ ¸ ¸ ¸
Unit Test ¸ ¸ ¸ ¸ ¸ ¸
Integration Test (Inter
¸ - - - - -
app)
Integration Test (Intra
¸ - - - - -
app)
System Test ¸ ¸ - - - -
Verification Test - - ¸ ¸ - -

Summary This adoption causes new challenges. Testing SOA

applications is still in its early stages and new techniques
are evolving as testing SOA based applications becomes
This paper attempts to bring all the risks, challenges and
more mature. It is anticipated that current GUI based touch
benefits of SOA under one umbrella so that test teams
point tools would also incorporate API based touch points
understand what they are up against. Adoption of SOA is
in their functionality to help testing teams conduct tests
on the rise as more applications are being introduced to
throughout the life cycle.
network inter connectivity.

AppLabs.com
App_WhitePaper_Approach_to_SOA_1v04 Page  © 2007 AppLabs