You are on page 1of 114

Concepts, Technologies,

and Best Practices


Beth Gold-Bernstein & Gary So
Integration and
SOA
next
2006
2
About webMethods
Get There Faster.
webMethods provides total business integration to the worlds largest corporations and
government agencies. webMethods fagship product suite, webMethods Fabric, leverages
leading SOA and BPM technologies and methodologies to deliver rapid ROI to our 1,300
customers around the globe. With webMethods, customers can take a process-centric
approach to their business problems, allowing them to re-use their existing IT assets,
dramatically improve business process productivity and ROI, and rapidly create competitive
advantage by making their business processes work harder for their company. webMethods
(NASDAQ: WEBM) is headquartered in Fairfax, VA, USA and has offces throughout the United
States, Europe, Asia Pacifc and Japan.
For more information on our company and our products, we invite you to visit our website,
www.webMethods.com
View our SOA webinar series www.webMethods.com/Events/Web/GetRealwithSOA
Learn more about us www.webMethods.com/About
Hear our vision www.webMethods.com/Vision
Meet our customers www.webMethods.com/Customers
View our products www.webMethods.com/Products
If you would like additional copies of the book, please email soabook@webMethods.com
About ebizQ
ebizQ The Insiders Guide to SOA and Integration
ebizQ is the leading community of business and IT executives, managers, architects and
developers focused on business integration and SOA. ebizQs analyst-centric multimedia
content includes interactive training, webinars, and analysis of industry trends and best
practices. You can access daily aggregation of in-house and industry-wide blogs, breaking
news, feature articles and white papers.
Learn more by visiting www.ebizQ.net and gain access to the following great content!
Webinars: www.ebizQ.net/webinars/
White Papers: www.ebizQ.net/white_papers/
Online Curriculum: www.ebizQ.net/trainingcenter
Blogs: www.ebizQ.net/blogs
CIO audio: www.ebizQ.net/executive_corner/cioaudio/
Buyers Guide: www.ebizQ.net/vendors
Bring the rest of your team up to speed. Attend our online companion course, a Managers
Guide to Integration and SOA: www.ebizQ.net/managersguide or call
1-866-55-EBIZQ (1-866-55-32497)
3
Tables of Contents
Chapter 1 The Business Imperative for Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Executive Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Challenges to Business Agility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
The Mandate for Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Integration and SOA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
The Evolution of Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Integration: A Business Imperative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
How to Use This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Chapter 2 Building the Business Case for Integration . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Executive Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Tangible and Intangible Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Areas for ROI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Defining Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Chapter 3 Common Integration Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Executive Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Data Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Enterprise Application Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Mainframe Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
B2B Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Single View of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Multichannel Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Composite Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Business Process Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Business Performance Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4
Chapter 4 Guide to Integration Technologies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Executive Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Application Integration Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Information Integration Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Interface Integration Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Composite Application Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Process Integration Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Business Optimization Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Compliance and Industry Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Management and Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Architecture and Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Chapter 5 Integration and SOA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Executive Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Why SOA? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
SOA Explained . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
The Role of Web Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Enterprise-Class SOA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Best Practices in SOA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Chapter 6 Best Practices for Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Executive Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Common Pitfalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Taking a Strategic Approach to Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Organizing for Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Responsibilities of the ICC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Creating an ICC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Critical Success Factors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5
Executive Overview
S
ometime around 500 B.C., the Greek philosopher Heraclitus said, Nothing endures
but change. While change may have been a constant since time immemorial, the
rate of change is accelerating far faster than ever before, and this is having a profound
effect on business. Business cycles are shrinking rapidly. The way business was con-
ducted even a decade ago is no longer acceptable if an organization wishes to remain
competitive. Organizations have had to change how they interact with customers, how
they manufacture goods, and how they are organized and managed.
Rapid changes are only possible when the organization itself is agile. The notion of
an agile business has long captured the imagination of business executives. The agile
business is able to embrace changes in market conditions, organizational structure,
and the regulatory environment without missing a beat. An agile business empow-
ers the management team to focus its collective acumen on delivering substantially
increased value to its stakeholders.
The desire for greater business agility is making integration increasingly important.
Agility is the combination of speed and adaptability: speed to bring new solutions to
market more quickly, and adaptability to new business requirements and competitive
pressures. This can only be achieved when business processes are able to be changed
nimbly. And that in turn is possible only when the underlying IT systems are integrated
flexibly to accommodate the speed of change required.
Integration also enables organizations to leverage existing IT investments to stream-
line their processes for greater efficiency and productivity. IT infrastructures need to
utilize existing applications and systems to the extent possible. This requires an inte-
grated infrastructure along with end-to-end visibility of the business processes across
disparate systems. It also requires an approach that supports the implementation of
business solutions from existing components. This is the idea behind Service Oriented
Architecture, or SOA. (See Chapter 5, Integration and SOA.)
The Business Imperative for Integration
1
6
Chapter 1
The Business Imperative for Integration
Challenges to Business Agility
Currently, business agility is constrained by a number of obstacles. These include inflex-
ible applications that cannot be easily changed or enhanced; non-integrated stovepipe
applications; inefficient business processes; a lack of visibility into business processes
and operations; challenges brought about by mergers, acquisitions, and regulations;
and a lack of alignment between an organizations strategic objectives and daily op-
erations. Integration technology can play a role in removing these obstacles. In fact, in
todays business climate, organizational growth may be inhibited without integration.
Integration technology is a key enabling factor in helping business and IT executives
transform their organizations; get to market more quickly; respond faster to business
opportunities, competitive pressures, and regulatory requirements; and differentiate
how their organizations do business. Business responsiveness has become a function
of an organizations ability to rapidly marshal the underlying IT systems in alignment
with business needs. This means leveraging existing assets while creating new busi-
ness functionality.
The Mandate for Integration
One only needs to look at the history of computing and the evolution of business soft-
ware to see why integration is a priority for most chief information officers (CIOs) today.
In the beginning, back-office operations were run by mainframe systems. These sys-
tems were built to optimize expensive computing resources, but not business agility.
They wereand remaindifficult to change and brittle to boot: Changing one thing
could easily break something else. The legacy systems still in place today generally run
an organizations core back-end operations in a robust and reliable way. However, they
are usually batch systems and are unable to respond flexibly to business needs for real-
time information or new functionality.
That unfulfilled business need gave rise to the first wave of distributed computing with
the emergence of minicomputers. Department managers could purchase them with
their budgets and select them based on their particular needs. Often, the availability of
7
Chapter 1
The Business Imperative for Integration
a packaged application that met 50% to 80% of the departments specific needs led to
the introduction of these new platforms.
Unix, PCs, and client server software further reduced the cost of department
systems, and they began to proliferate throughout organizations. Low-cost desktop
productivity and development toolssuch as Microsoft Excel, Access, and Visual Ba-
sicsupported ad hoc solutions, which also proliferated and often became more stra-
tegic than they were originally designed to be. Individual business units established
their own computing facilities and application development capabilities, in effect set-
ting up shadow IT organizations. This gave them the desired level of agilityat least at
firstto respond more rapidly and independently to new business needs. However, it
also led to islands of automation: hundreds of applications spread across the organiza-
tion, many of them on desktops. These applications used organizational information
and were in turn used to report on business operations. But they were not under orga-
nizational IT management and were certainly not integrated consistently.
The emergence of the Internet and the associated rush to e-business further punctu-
ated the need for organizationsand, by association, ITto become more responsive
to market dynamics. But with different departments launching their own initiatives
and often duplicating one anothers work, the result was reduced visibility and control,
along with reduced economies of scale.
The widespread adoption of distributed systems often meant that large organizations
had multiple platforms running hundreds of applications that managed similar infor-
mation through different portions of various business processes. Unfortunately, these
applications were not designed to integrate with one another, so the organizations
had to find ways to keep the information in synch across the systems. Rekeying the
dataa means of last resort, but all too often the approach takenwas slow, resource-
intensive, and prone to errors that could be costly to trace and resolve. So an easier way
to integrate the disparate stand-alone systems was needed.
Another problem was that each packaged application was designed to focus on spe-
cific department processes. Business agility requires the optimization of business pro-
cesses end-to-end. This includes improving the process for initiators such as customers,
8
Chapter 1
The Business Imperative for Integration
partners, and suppliers as well as people in different business roles, and developing
systems that support different parts of the business.
While some emerging technologies look for a business problem to address, integration
was a business problem long before it was a technology. Packaged software systems
were far from being turnkey solutions. The integration costs of implementing the typi-
cal enterprise resource planning (ERP) system could be three to five times the cost of
the software. The problem was that the integration involved point-to-point hand-cod-
ing. This required an understanding of the application program interfaces (APIs) of all
the systems to be integrated and a high level of expertise. It also took a great deal of
time and was inflexible to change. The number of interfaces rose exponentially with
the number of systems being integrated. Such integration spaghetti was difficult to
manage and change, and upgrading to a new version of any of the applications meant
going through the process all over again.
This brings us to the present. Although the current state of affairs is largely the result of
short-term business decisions, it is ultimately viewed as an IT problem. Thats because
business change is intimately tied to the underlying IT systems flexibilityor lack
thereof. As markets move quickly and new opportunities and competitors emerge, or-
ganizations are increasingly challenged to close the gap between their business needs
and the lack of flexibility in their IT infrastructures. Perhaps the biggest obstacle to
closing this gap is that change has to be implemented while the core IT operations that
support the business continue to function smoothly and seamlessly. The challenge is
akin to upgrading the wings of a plane while its in flight. This is where Service Oriented
Architecture enters the picture.
9
Chapter 1
The Business Imperative for Integration
Case Study 1-1
Company: RBank
Company Description
RBank is a pseudonym for the retail banking subsidiary of one of the nations largest financial
services companies. It provides investment management, retail and commercial banking, re-
tirement, consumer finance, and investment banking products and services to individuals and
organizations throughout the United States and, for certain businesses, internationally.
Business Challenges
RBank had more than 60 different customer response systems and applications, and all
customer service requests had to be routed to specific lines of businessa process involving
several stepsbefore the customer was directed to the appropriate representative. RBank
also faced challenges complying with an increasing number of federal laws. The Patriot Act,
for example, expanded anti-money-laundering requirements through provisions to deter illicit
financial activities that support the financing of terrorism. RBank needed to ensure that it was
submitting the appropriate customer information to its agency in a timely manner to meet the
compliance requirements.
How Integration Addressed the Challenges
RBanks enterprise architecture division adopted an SOA and the webMethods Fabric inte-
gration platform to build applications faster and cheaper. RBank representatives now have
comprehensive information on all customer interactions with the bank on one screen, making
the customer experience much more positive. Using the integration solution, RBank employees
can make changes to customer information, such as address updates, in one place and have
them propagated and synchronized across all customer systems. RBank also used the new
integration solution to automate previously manual processes, including customer loan ap-
plications. Applications are now routed through the integration platform to RBanks financial
systems for processing, and decisions are returned to the customer in real-time.
The SOA approach and integration platform solution has also played a critical role in compli-
ance efforts. RBank can now connect directly to its agency to transmit all the required cus-
tomer information and manage the verification process in real-time.
Bottom-Line Business Benefits
The 360-degree customer view has enabled RBank to customize services to customers as
well as to cross-sell and up-sell offerings, improving customer satisfaction and growing sales
opportunities. By automating processes and eliminating error-prone manual steps, RBank has
realized substantial cost savings.
10
Chapter 1
The Business Imperative for Integration
Integration and SOA
At the core, SOA is about creating systems out of standard building blocks. The concept
is not new. Many of the principles underlying SOA, such as isolating functionality to
promote reusability, are long-standing best practices. In the past, however, the adop-
tion of SOA was hindered by a lack of widely accepted standards for putting together
these building blocks. Now, the pressing demands of businesswhich necessitate in-
creased system flexibility and adaptabilityhave inspired nearly all organizations to
align on a single set of standards for SOA: Web services.
While SOA involves much more than Web services, standardization is largely respon-
sible for removing the barriers to SOA and fostering a future where systems can be
more easily assembled and incrementally modified. Today, in the face of growing com-
petitive pressure and the accelerating pace of business, organizations realize that they
run a risk if they do not move toward SOA.
So how are SOA and integration related? In fact, they are highly complementary.
SOA is inherently about a distributed architecture, with systems that span computing
platforms, data sources, and technologies. A distributed architecture requires integra-
tion. By standardizing how systems interoperate, Web services simplify the task of in-
tegration. Web services alone, however, do not suffice. Organizations need an evolu-
tionary approach to SOA that incorporates legacy (non-Web-services-based) systems.
Integration software provides the bridge between the legacy systems and SOA, allow-
ing organizations to leverage existing software assets while managing their transition
to SOA.
Integration solutions also contribute mature technologiessuch as messaging, rout-
ing, data translation and transformation, and event managementalong with organi-
zational disciplines that are necessary for full-fledged, enterprise SOA. (See Chapter 4,
Guide to Integration Technologies.) Moreover, integration capabilities such as business
process management (BPM) and business activity monitoring (BAM) allow organiza-
tions to realize a higher level of business productivity from SOA by enabling the opti-
mization of business processes and the alignment of strategic objectives with opera-
tional actions. In short, integration should play a central role in any organizations SOA
strategy.
11
Chapter 1
The Business Imperative for Integration
The Evolution of Integration
An organizations IT history remains its present. Organizations seldom have the luxury
of starting with a clean slate. With the vast investments that organizations have made
in systems and infrastructures, CIOs have an overriding financial imperative to leverage
existing resources to support new business requirements. In addition, it is very risky to
rip out and replace complex systems that serve mission-critical business operations.
So new or modified business services need to be delivered rapidly without affecting
current operations. They also need to maximize the use of in-place assets, with all their
associated constraints.
Integration software has emerged as an effective and viable approach to bridging
organizational silos. But so far organizations have had only modest success in truly
leveraging their existing in-place application assetswhether legacy or packaged ap-
plicationsto support new business initiatives. To successfully deliver business agility,
organizations must evolve their definition of integration and its application to their
business objectives. No longer only about integrating systems, the role of integration
now also encompasses quickly assembling new solutions for the business, and moni-
toring and optimizing an organizations business processes and operations.
Integrate. To manage the full scope of the business and to increase efficiencies,
organizations must integrate previously autonomous business processes and
their underlying applications. Integration software on its own is insufficient.
At best, it will only allow organizations to streamline their existing processes.
Business agility requires the ability to extend existing IT assets as reusable ser-
vices. This is the concept behind SOA. Not only does it enable organizations
to leverage the investments they have already made, but it promotes reuse,
with all the associated benefits of reduced cost and improved quality. The new
world order, therefore, demands an SOA-based approach to integration.
AssembIe. Putting in place an SOA-based integration foundation enables the
rapid assembly of new solutions from the catalog of available services. This
ability to decompose monolithic applications into reusable business services,
and then to recompose them as needed, contributes directly to business agil-
ity by enabling organizations to rapidly change and assemble new business
processes to support changing business priorities. When combined with BPM
4
4
12
Chapter 1
The Business Imperative for Integration
to enable processes to be composed from existing building blocks, SOA allows
operational changes to be delivered faster, cheaper, with a higher degree of
quality and with a higher return on assets.
Monitor. SOA also makes it uniquely possible for organizations to monitor ev-
ery aspect of their key business processes and gain insight into how they can
optimize these business processes. By using the services within an SOA-based
system as measurement points, organizations can gain unprecedented visibil-
ity into the metricssuch as order processing times, average order amount,
and so onthat reflect the operational and business performance of a pro-
cess.
Dptimize. Then, by comparing these real-time performance metrics against
previously defined key performance indicators, organizations can create a
feedback loop that facilitates iterative improvement of their processesfor
example, by helping identify and reduce the bottlenecks in an order-to-cash
workflow. Again, SOA plays a role by simplifying and speeding an organiza-
tions ability to make the desired business process changes, or implement
the necessary system solutions, to drive these improvements. This ability to
iteratively improve the way the business operates ultimately results in greater
business process productivity for the organization.
Integration: A Business Imperative
Business agility is becoming a strategic goal to organizations in every industry, and
in both the private and public sectors. The bottom line is that IT must be able to re-
spond more quickly to changing business needs, opportunities, and threats. It is a mat-
ter of survival. As discussed in this chapter, agility is enabled through an integration
infrastructure implemented with SOA principles. The glue that binds together all these
business goals and implementation strategies is integration. Organizations need to
create core competencies around integration and SOA, much the way they did with
databases and data centers in the 1980s and 90s. Integration is nothing less than a
core requirement for enabling business agility.
4
4
13
Chapter 1
The Business Imperative for Integration
Case Study 1-2
Company: Electro
Company Description
Electro is a pseudonym for a leading global distributor of electronic components and comput-
er products. With more than 100,000 customers and 300 suppliers, it markets, distributes, and
adds value to a wide variety of electronic components, enterprise computing products, and
embedded systems. The company is positioned between multiple tiers of customersvalue-
added resellers (VARs), contract manufacturers, and end usersand a large supplier base.
Business Challenges
One of Electros major suppliers mandated that its software customers buy through the
distribution channel. This forced multiple VARs to quickly choose a distribution partner for
order fulfillment because they could no longer buy directly from the manufacturer. Most of the
VARs used EDI for business, and they needed to select and migrate their current EDI ordering
processes to a new partner within 60 days. Electro saw the opportunity to provide drop-in
integration capability and capture the substantial revenue stream. To capitalize on the oppor-
tunity, however, it needed to quickly put in place a solution that would connect the two ends
of the supply chain by allowing each partner to leverage its existing EDI solution.
How Integration Addressed the Challenges
Using an out-of-the-box EDI solution from webMethods, Electro accelerated the creation of an
EDI framework so potential VAR customers could connect for fulfillment. The company lever-
aged its existing webMethods integration infrastructure, which contained many key integra-
tion points and core services necessary to construct a solution rapidly. The EDI modules were
installed into the infrastructure and utilized for document translation and data transformation.
The resulting framework was reusable and enabled Electro to provide drop-in integration
capability to the targeted VARs.
Bottom-Line Business Benefits
Electro created more than $200 million in new, incremental revenue with a minimal invest-
ment ($50,000). With the new EDI integration solution, it quickly added a number of key
VARs to its customer list. In addition, the project yielded productivity gains of 30% to 45%.
This included an estimated 60% of reuse from existing integration assets, which sped time-
to-market. The solution enabled Electro to not only target new customers and their potential
revenues but also create entirely new business models that otherwise could not have been
addressed.
14
Chapter 1
The Business Imperative for Integration
How to Use This Book
This book is designed as a primer for those who need to understand integration and
how to apply it in their organizations to achieve business goals. The book is organized
as follows:
Chapter 2, Building the Business Case for Integration, more deeply defines the
business benefits and return on investment that can be achieved through in-
tegration. This chapter helps you build a business case for integration proj-
ects.
Chapter 3, Common Integration Scenarios, helps you identify the business ini-
tiatives that can be driven by integration technology.
Chapter 4, Guide to Integration Technologies, further explains the technolo-
gies available and how they relate to different business solutions.
Chapter 5, Integration and SOA, defines the relationships between SOA and
integration, and shows how to optimize your IT infrastructure to enable busi-
ness agility.
Chapter 6, Best Practices for Integration, offers guidelines to ensure that enter-
prise integration becomes a core competency in your organization.
Throughout the book, we have provided case studies that illustrate the benefits of in-
tegration in specific situations. Although we have changed the companies names to
maintain anonymity, they are actual organizations that have used integration products
to achieve their goals.
In closing, remember that integration is not an end point. It is not a single technology
that comes in a shrink-wrapped box, nor is it even a single project. As you read this
book, it will become increasingly clear that integration is a journey. Success requires
technology, process, and people. We sincerely hope that this book will be a valuable
guide on your integration journey.
4
4
4
4
4
15
Executive Overview
M
any past investments in infrastructure technology have failed to deliver real
business benefits and return on investment (ROI), making some organizations
leery of investing in integration software. However, integration is a key component in
the creation of a more agile enterprise. It can help businesses streamline operations,
provide better customer service through access to more complete information, and
enable faster response to competitive threats. Progressive organizations are leverag-
ing integration strategically to fundamentally differentiate their products and services
and to gain competitive advantages that drive revenue growth. In IT, integration can
reduce the cost and time required to implement new solutions while improving ef-
ficiency. More intangible benefits include the ability to enhance the organizations
brand value by enabling modern, consumer-oriented solutions and a higher degree
of service quality.
Business cases for integration should reflect a combination of tangible returns in hard
dollars and more intangible returns that, although difficult to enumerate, offer sustain-
able business value.
Organizations also need to understand that integration was a requirement long before
packaged integration middleware solutions became available. Before the availability
of off-the-shelf integration technology, implementing a software package with inter-
faces to other applications, or combining systems because of a merger or acquisition,
typically meant hard-wiring the systems together in a point-to-point fashion using
hand-coded interfaces. With this type of integration, the number of interfaces involved
rises exponentially for each additional system being integrated. Furthermore, the pro-
prietary nature of each interface makes it impossible to leverage existing work when
adding new systems. And, because they are hand-coded, the interfaces tend to be
brittle and resistant to change. As a result, upgrading a software package can require
integration costs several times the purchase price of the package. The net result is that
Building the Business Case for Integration
2
16
Chapter 2
Building the Business Case for Integration
a significant percentage of IT organizations budgets are being consumed by the high
ongoing maintenance and support costs for their home-grown integration solutions.
When making a buy vs. build decision for integration software, organizations need to
factor in the total cost of ownership. In nearly all cases, off-the-shelf integration soft-
ware solutions deliver a better ROI than internally developed alternatives. Off-the-shelf
integration software is also typically more functional and robust, since the vendor can
invest more R&D dollars into development and provide support for the software.
This chapter helps you to identify the benefits of integration and to build a business
case and calculate the ROI you can expect from your investment in a packaged integra-
tion solution.
Tangible and Intangible Benefits
Because of the complexity of integration and the mixture of tangible and intangible
benefits, it is rarely possible to build an integration value case that is purely financially
based. Instead, the overall business case for investing in an integration infrastructure is
likely to have three parts:
Dollars representing the direct cost benefits of integration
Dollars representing the indirect benefits of integration
Soft or intangible factors that must be considered as additional weight in any
judgment
Direct cost benefits from an integration solution represent actual dollars saved, such
as reduced development costs for new projects, reduced maintenance costs, reduced
errors and the cost of fixing errors, and reduced resource needs.
Assigning estimated values to indirect benefits is more challenging but equally es-
sential. While indirect benefits may be difficult to quantify in the short term, they often
provide the largest long-term benefits. The key here is to ensure that the affected busi-
ness units agree with the assessed benefit value. For example, if improved integration
4
4
4
17
Chapter 2
Building the Business Case for Integration
delivers a more efficient just-in-time process for a manufacturing company, one ben-
efit will be reduced inventory. On this basis, the inventory manager can provide a value
estimate. It should be possible to validate this figure as the project is deployed.
The third part of the justificationsoft or intangible benefitsis the most difficult to
address, but it is nevertheless an important part of the consideration. Suppose the in-
tegration project delivers a more responsive customer self-service system. One benefit
should be improved customer satisfaction and, hence, customer loyalty. Though this
may be a key organizational goal, assigning a dollar value to the benefit is difficult. Or
suppose the integration project is part of fulfilling legislation, such as Sarbanes-Oxley
requirements. Again, it is hard to place a dollar value on this, though noncompliance
would have significant implications for the business.
Soft or intangible factors make evaluating integration projects an art as well as a sci-
ence, in that a value judgment is required in addition to hard numbers. Nevertheless,
decision makers can still obtain a meaningful broad-brush feel as to whether the in-
vestment makes sense. And later refinement will make the case more accurate, even if
some benefits remain intangible and subjective.
Figure 2-1. Examples of benefits to consider
in an integration investment decision
18
Chapter 2
Building the Business Case for Integration
Areas for ROI
Integration is a key infrastructure technology for delivering a new breed of more agile
enterprise applications, but it does present challenges for justifying investments. First,
business investments are usually made on a project-by-project basis, while infrastruc-
ture investments span multiple projects. It is often easier to justify investments across
projects if an organization takes an enterprise approach to integration and the recom-
mendations come from an Enterprise Architecture Group or Integration Competency
Center (ICC) within the Architecture Group. To justify integration infrastructure invest-
ments, look for both IT and business benefits, as well as benefits to the organization as
a whole.
IT BenefITs
IT can benefit from integration through decreased costs and increased agility. Cost re-
duction can be measured in real dollars. Increased agility is more difficult to quantify,
but it can provide potentially much larger long-term benefits and ROI.
Direct Cost Benefits
The direct cost savings for IT through integration include reductions in three areas:
Development and implementation costs
Skills requirements
Operating costs
Integration software provides a single interface point to individual applications. Instead
of programming to a different application program interface (API) for each application,
integration platforms and Web services provide a standard interface to the integra-
tion software that acts as a broker. This drastically reduces the number of interfaces
requiredand, thus, the number of interfaces that need to be developed, tested, and
maintained. The savings can be calculated by first determining the cost of develop-
ing interfaces for each application that needs to be integrated, then adding the cost
of maintaining those interfaces, and finally comparing the total cost with the cost of
purchasing the vendor-developed and maintained adapters.
4
4
4
19
Chapter 2
Building the Business Case for Integration
Case Study 2-1
Company: ScandAir
Company Description
ScandAir is a pseudonym for an Iceland-based airline that carries 1.5 million passengers each year
on its fleet of Boeing 757 and 767 jets. ScandAir has sales offices and flight destinations across
Europe and North America, and serves tourist and business traffic to Iceland as well as interna-
tional traffic via its hub in Reykjavik.
Business Challenges
Internet bookings of airline tickets were rising, and ScandAirs Web sites felt the impact. This
evolving business channel had the potential to reduce costs and improve efficiencies, but it was
having the opposite effect. Each online sale required an average of 11 minutes of administrative
back-office support, negating any business benefits possible from e-commerce.
At the same time, ScandAirs IT department was evaluating the need for an integration platform
to help solve the increasingly complex web of enterprise applications that powered the company.
ScandAirs financial system, for example, interfaced with more than 30 applications. Managing
this had become increasingly challenging. The company was looking for an area to test a new
enterprise integration solution that would demonstrate a measurable ROI. The automation of its
online airline ticketing process provided the ideal pilot project.
How Integration Addressed the Challenges
ScandAir wanted to use enterprise integration to deliver a fully automated
e-ticketing system that would eliminate the time spent on its current manual processes. Using a
local integration partner and components from the webMethods Fabric suite, ScandAir rolled out a
successful initial pilot in less than three months that automated the back-office process associated
with Internet bookings in the Reykjavik sales office.
After this projects success, the effort quickly expanded to include all bookings out of Reykjavik.
Then the model was implemented in each ScandAir sales region. The project included developing
adapters for the various back-office systems, including the TPOS credit-card clearing system and
the AMADEUS flight reservation system. These adapters were then used across the organization
for other purposes, saving development time and effort.
Bottom-Line Business Benefits
Since ScandAirs first implementation in 2003, more than 150,000 bookings have been completed.
The automation has saved more than 27,500 hours of manual effort when compared with the
previous processes. In addition, standardization has brought new efficiency gains in business pro-
cesses and IT. ScandAir now processes its invoices electronicallysomething it did not originally
foreseeand this has produced additional savings as a result of reduced postage and paper use
and reduced storage space for the paper invoices.
20
Chapter 2
Building the Business Case for Integration
Integration software also reduces skills requirements because, typically, the integration
software shields developers from having to deal with many underlying technicalities.
An integration suite offers a standard approach that benefits not just the implemen-
tation process but also operations, further reducing the requirement for special skills.
This may eliminate the need to hire or contract with expensive specialist personnel.
Additionally, since a vendor is now responsible for the integration mechanism be-
tween components, it can be relied on to adapt to new technology advancements (for
example, ensuing that connectivity to new versions of an application such as SAP or
Siebel is maintained).
Finally, integration software can decrease operating costs by reducing transaction er-
ror rates. It automates the transfer of information into systems and eliminates the need
for rekeying, which reduces manual errors and thus offers significant ROI. Finding and
fixing errors increases the cost of a transaction by orders of magnitude.
In fact, in ROI studies conducted by ebizQ, the reduction in error rates alone could
quickly justify the cost of the system. For example, by using integration software, a
major North American transportation company eliminated 70,000 to 80,000 hand-pro-
cessed transactions, reducing the cost per order from $25 to $5. This amounted to a
$1.6 million annual savings. In another example, a high-tech manufacturer using Ro-
settaNet reported an 85% reduction in error rates and achieved a 100% ROI in the first
year. And in 1999, 81% of Ciscos orders were placed online with 98% to 99% accuracy,
representing $11.7 billion in revenue annually. Organizations that can quantify error
rates in their current systems processing are often able to assign a significant value to
error reduction.
Indirect Benefits
The biggest indirect cost benefit for IT comes from the use of technology and solu-
tions. For example, when Hurricane Katrina hit the Gulf Coast in August 2005, a North
American insurance company wanted to make accommodations for its policyholders
affected by the hurricane, such as providing a grace period on premiums. IT had already
built a collection of Web services to access its mainframe systems in support of another
large project. Now it was asked to quickly build Web pages for the custom site. Even
21
Chapter 2
Building the Business Case for Integration
though the high-level Web services developed for the other project were not suitable
to support the new business requirements, a single member of the IT team was able
to quicklywithin a day or tworecombine lower-level worker services to create the
necessary Web services for the Katrina site. This impressed the business managers and
was recognized as a Service Oriented Architecture (SOA) success story up to the CIO
level.
Soft Benefits
The soft benefits of integration for IT include increased responsiveness to the business
and an increased perception of IT as a business enabler. New business processes can
be deployed more quickly because of the reduced effort and the reusability offered by
integration. Further, as new technologies such as Web services take hold, a coherent
integration architecture delivers an enhanced level of system flexibility, enabling these
technologies to be embraced and exploited faster and cheaper. And, of course, the use
of vendor-supported integration technology means that responding to new technol-
ogy challenges is the responsibility of the vendor rather than the IT department.
BusIness BenefITs
The business benefits of integration include enhanced process efficiency and produc-
tivity, increased operational effectiveness, and improved competitive performance. En-
hanced efficiency and productivity often translate into direct cost savings. Increased
operational effectiveness often provides many indirect benefits. Improved competitive
performance can be the most valuable benefit to the organization, but also the most
difficult to quantify.
Direct Cost Benefits
Integration software improves efficiency and productivity in a number of ways. First, in-
tegration can increase levels of automation and reduce human intervention. Examples
include reducing order times by removing manual order reentry at different stages
of the processing cycle, cutting the percentage of failed funds transfer requests due
to input errors, and improving the speed of information sharing between different
22
Chapter 2
Building the Business Case for Integration
departments. Cost reductions would result from decreased staffing levels due to the
increased level of automation or the use of a more productive process. In addition, pro-
cess efficiency may also allow a reduction of inventory through just-in-time processes
that rely on a high degree of automation. This becomes even more feasible when the
process efficiency is extended to encompass suppliers and distributors, and it can have
the side effect of reducing procurement costs as well.
When business process management (BPM) functionality is layered on top of basic
integration capabilities, process efficiency and effectiveness are increased, and this can
have very quantifiable bottom-line results. In the 1950s, Dr. W. Edward Deming, a math-
ematician and statistician, showed that variations created at every step in a process can
cause defects, which are expensive and time-consuming to remediate. Demings work
with Japanese manufacturers was directly responsible for the low-cost, high-quality
goods that competed favorably against American manufactured goods.
Demings work gave rise to a number of initiatives to manage and measure the effi-
ciency and effectiveness of business, including Total Quality Management (TQM), Bal-
anced Scorecard, Six Sigma, and ISO 9001. Organizations that adopted these practices
realized measurable results. In fact, some integration software products on the market
today enable organizations to apply these quality management and measurement
practices to their business processes in the act of integrating them.
Indirect Benefits
Operational effectiveness creates more indirect benefits. Integration can help make
a business more effective by improving business intelligence and its dissemination.
For example, the ability to pool information about a particular prospect from different
parts of the business may make cross-selling or up-selling possible. Or the ability to
provide comprehensive feedback on product performance directly from customers to
Engineering may enable the development of new processes to improve product qual-
ity. Assessing the ROI depends on the specifics of the situation; for example, benefits
might be realized by reducing the cost of sales or increasing sales volumes.
23
Chapter 2
Building the Business Case for Integration
Case Study 2-2
Company: Annuate, Inc.
Company Description
Annuate, Inc. is a pseudonym for an Australian super-annuation fund administrator. The company man-
ages 5 million member accounts and more than 340,000 employer accounts. It provides administrative
services such as bulk contribution processing, fund take-ons, benefit processing, insurance and claims
management, member communications, and the management of member inquiries.
Business Challenges
For several years, Annuate had been vying for a major new account, the Australian Retirement Fund
(ARF), with more than 600,000 members. Although Annuate had always been competitive in the ten-
der process, it learned that its business process applications did not meet ARFs requirements. Annuate
identified its technology architecture as the main reason for losing the business. And it realized that
to be competitive it would need to build a world-class technology infrastructure founded on a solid
integration strategy that would enable new systems to be deployed while leveraging existing legacy
systems.
How Integration Addressed the Challenges
Annuate deployed webMethods Fabric to integrate with a new clients applications and facilitate real-
time dissemination of information across the enterprise. This new enterprise integration infrastructure
allowed it to win the ARF account and to merge 600,000 ARF records in the course of a six-month
project that established the funds records on the core administration system, provided data confir-
mation and business rule coding, and made the other applications work off its Oracle-based member
administration system.
As part of Annuates integration efforts, it adopted an SOA platform that enabled the company to
reuse many of the services it created. Annuate can now expose Web services through the integration of
applications such as Web sites and internal administration systems, and can repurpose the components
of its existing legacy systems, further leveraging existing technology.
The integration also allows Annuates clients to view their portfolios online, get daily estimates of their
account balances, view regular e-newsletters, and learn about investment changes. These online tools
complement the information provided on member statements. The real-time updates of information
ensure accurate information to customers and across the enterprise.
Bottom-Line Business Benefits
The ARF win was a direct outcome of Annuates clear enterprise integration strategy. The implementa-
tion of the enterprise integration solution has allowed a number of back-end systems to interact and
provide a complete administration service for the client. As a result, Annuate has already seen a 30%
to 40% reuse of interfaces and services created, as well as reduced maintenance costs. The entire solu-
tion also enables IT to better support Annuates business strategy.
24
Chapter 2
Building the Business Case for Integration
Soft Benefits
Being able to serve customers needs faster and cheaper improves the businesss
chances of beating the competition. In addition, the ability for process changes to be
implemented faster at the operations level, through underlying IT mechanisms that
are more flexible to change due to integration, enables business operations to respond
to competitive action more quickly. Once an integration infrastructure is in place, it
reduces the cost of implementing change. It could also lead to increased revenue by
enabling a business unit to lead with new differentiating initiatives. These competitive
gains can be evaluated in monetary terms by assessing the likely market share increase
that results from operational efficiency and speed of reaction.
OrganIzaTIOnal BenefITs
Finally, there are the returns at the organizational level. Of course, the two areas already
describedIT benefits and business benefitsalready have an effect on the overall
organizational performance. But this section concentrates on areas that apply more
specifically to the organization as a whole, reflecting the brand value across all aspects
of the business. These benefits include:
Compliance with government and industry regulations
Customer satisfaction
Exploitation of the organizational knowledge base
Access to new opportunities
Most organizational returns can be categorized as indirect or soft benefits. Benefits may
result even when integration itself is secondary to the underlying business require-
ment. For example, many organizations are finding unexpected benefits from their
compliance initiatives. The ability to integrate different information sources improves
the understanding of organizational operations and procedures. Moreover, particu-
larly with some of the more recent forms of legislation, such as Sarbanes-Oxley and
Basel II, it enables organizations to build sophisticated audit trails of their operations.
This has the beneficial side effect of providing organizations with a better window into
4
4
4
4
25
Chapter 2
Building the Business Case for Integration
how their business runs. In reality, any integration project that plays a key role in meet-
ing compliance mandates is likely to be justified on a nonfinancial basis (indeed, it is
often a case of organizational survival).
Integration plays a significant role in customer satisfaction. It can provide a complete,
organization-wide single view of the customer in real (or nearly real) time. This picture
may include current product sales, product experiences, reported issues, outstanding
deliveries, and many other facets of customer-related information. This capability al-
lows the organization to provide a much higher level of service to the customer, for
example through its call centers or help desks. Customer satisfaction leads to customer
loyalty, which many organizations can relate to financial performance, for example by
comparing the expense of securing new customers with the cost of retaining existing
ones. Regardless of how meaningfully the benefit can be quantified, customer satisfac-
tion is an important benefit of integration.
Integration can also bring together knowledge from all of an organizations business
units, making it available for sharing and thereby increasing the overall value of the
organizational knowledge base. This can have paybacks in improved organizational
efficiency, more rational organizational structures, and a more effective workforce. For
example, providing employee portals that offer information about organizational pro-
cesses and procedures, such as health and safety, may simultaneously reduce educa-
tion costs for new employees, help improve employee morale, and achieve compliance
with legal requirements.
Finally, integration can link together the entire value chain, both within the organiza-
tion and outside to partners and even customers to enable to new opportunities. This
wide-reaching integration makes it possible to introduce new business products or
initiatives. In fact, many organizations are using integration strategically to offer new
services to new markets.
Defining Costs
To evaluate ROI, the benefits that accrue from the integration project must be balanced
26
Chapter 2
Building the Business Case for Integration
against the costs incurred. Broadly, IT costs tend to be grouped around the provision
and support of the IT changes required to implement the desired operational changes.
In contrast, business unit and organizational costs tend to be expressed in terms of
the changes required to adapt to the new operational processes and procedures. It is
important to remember that both are relevant, although the costs from the business
side of the organization are harder to evaluate. Organizations that forget the impact to
business personnel generally suffer as a result of the oversight.
IT COsTs
IT costs resulting from an integration project generally fall into three categories:
Infrastructure costs
People costs
Services
Infrastructure costs include the expected software license and maintenance costs.
These cover the integration solution selected, any new application packages, and any
other new pieces of infrastructure software required. It may also be necessary to ac-
quire hardware to handle the additional capacity generated by the solution or to pro-
vide local processing power as needed. While the initial infrastructure costs may be
high, they are easier to justify when amortized across multiple projects. This is another
good reason to take an enterprise approach to integration.
Clearly, a wide range of people costs must be included in any evaluation. These span
the traditional range of project development, encompassing planning, design, devel-
opment, testing, deployment, and support. At the planning and design levels, it is im-
portant to have both IT and business skills, since integration projects deal with both
aspects (for example, business processes have to be defined or modified to produce
the required business return, and this must be explained to the IT team). In addition,
business domain knowledge may be required to help IT understand the business stan-
dards, such as RosettaNet, that need to be taken into account in the solution.
4
4
4
27
Chapter 2
Building the Business Case for Integration
Case Study 2-3
Company: DNA Corp.
Company Description
DNA Corp. is a pseudonym for a life sciences company that specializes in research technolo-
gies, diagnostic products, and targeted therapeutics. Pharmaceutical and biotechnology com-
panies use its products to discover and develop new drugs more effectively. DNA Corp. has an
installed base of approximately 180,000 instrument systems in nearly 100 countries.
Business Challenges
DNA Corp. builds customized assays for its customers. This is a complicated process that
involves performing detailed analyses of DNA sequences. Traditionally, a customer uploaded a
file containing the sequence to DNA Corp.s e-commerce Web site. The order was then routed
via FTP through DNA Corp.s back-office systems, which included 13 enterprise systems from
order creation through accounts receivable. This process was error-prone, and recovering from
any problems that developed was labor-intensive. The development time for each new assay
could take 12 to 16 weeks. DNA Corp. needed to streamline the process for building custom-
ized assays to improve its competitive position and reduce costly errors.
How Integration Addressed the Challenges
DNA Corp. wanted to separate the process of managing and accessing the information about
the assays from its order entry system in SAP. To streamline the process, it adopted an SOA-
based enterprise integration solution from webMethods. The new system allows customers
to define assay requirements through a Web front end, which then passes an XML message
to the design engine. The engine attempts to produce an assay design to match customer
requirements. The customer can validate the design and then choose to have the assay built
based on optimizing cost, accuracy, repeatability, or other factors. These additional parameters
determine the process sequences used. Externalizing the assay information from SAP provides
greater flexibility and reuse, and reduces the overall time to build a customized assay.
Bottom-Line Business Benefits
The new enterprise integration solution has had a direct impact on DNA Corp.s bottom line.
By integrating the 13 back-end systems involved, it reduced the development time for a new
assay type from 16 weeks to two weeks. This enabled DNA Corp. to reduce its order fulfillment
errors by 20% and decrease overall order time from three weeks to less than five days.
28
Chapter 2
Building the Business Case for Integration
The design phase needs to consider not just the technical architecture and approach
but also the operational processes and workflows necessary to support the solution.
Development resources are required to build the interfaces to the components be-
ing integrated, to modify the relevant IT infrastructure, and to supply essential mecha-
nisms, such as audit and problem determination trails. The integration software itself
has to be configured for the operating environment.
The test phase may require specific hardware and software, for instance to validate
performance requirements and to provide a thorough quality assessment of the new
software implementation. Personnel are required to carry out the testing and to resolve
and rework problems as required. Finally, deployment and support costs must include
items such as help desk and operations staffing and training, quite possibly with indi-
viduals who have business as well as technical skills, to ensure successful operations.
Throughout the range of project activities, it may also be necessary to consider third-
party service costs. Some organizations may decide to utilize systems integration ex-
perts to provide valuable guidance in the planning and design phases of the project.
Parts of the project may also deal with a specialized area of technology knowledge,
and in these cases it may be more financially viable to contract for the skills rather
than retrain existing IT staff. Examples include retaining someone to integrate a par-
ticular software package or someone skilled in the particular integration technology
selected.
BusIness ImpaCTs and COsTs
The three categories most likely to be taken into account in the area of business costs
are:
Resources to support project implementation
Education and training of end users
Operational risks
4
4
4
29
Chapter 2
Building the Business Case for Integration
The first category has already been described in the previous section. Because integra-
tion projects span business and technical disciplines, resources to support planning,
design, implementation, and deployment activities are required. These resources are
necessary to ensure not just that the right solution has been built but that the new
solution is deployed in a controlled fashion to minimize disruption and business risk.
The education and training of the end-user community is another important cost to
consider. If the integration project is simply automating some back-room operations,
then the end-user experience and procedures may remain unchanged. But if new or
changed business processes are being implemented, retraining is an essential element
to be considered. Additionally, if the integration project spans business partners as well
as internal departments, the partners may also need education.
The area of operational risks covers aspects such as the loss of productivity during the
retraining and changeover to the new process, and potential service disruptions due
to technical problems or user errors experienced as the solution becomes fully estab-
lished. Since it is difficult to determine the impact of these issues and their associated
cost, decision makers may simply want to note them as part of their overall evaluation
criteria.
Best Practices
The following guidelines will assist you in determining the potential ROI for your inte-
gration project.
Know what you are measuring. Obviously, when measuring the success of any project,
you must first know what you are measuring. Set your objectives for the ROI audit. Be-
fore you can measure improvements, you need to know what your current processes
cost. Take baseline measurements before you begin the project. Measure the initial
performance levels of the processes under consideration for change or optimization
to build a baseline for future comparison. If you have already defined critical success
factors, you can do a baseline measurement of them. We recommend budgeting one
to two weeks for the initial estimate. Then determine how often you want to measure
your progress against your goals. We recommend a quarterly or semiannual cycle.
30
Chapter 2
Building the Business Case for Integration
Get executive commitment. Identify an ROI audit sponsor. This person is usually the
senior manager of the business unit affected by the implementation. Ideally, he or she
should be able to sell the audit findings throughout the organization and be willing to
defend the logic, assumptions, and results confidently.
Focus on improving business processes. Ultimately, improving business processes will
yield the highest return on investment. To begin, identify some initial processes that
are the low-hanging fruitripe for automation and integration.
Start small and lay the foundation for future enhancements. Nothing succeeds like suc-
cess. Make sure the first projects have a high probability of succeeding. Keep the scope
relatively small but still pertinent to the business.
Avoid analysis paralysis when doing the ROI study. Set a number of hours or days com-
mitted to ROI analysis, and stick to it.
Build a credible business case for integration. Do your homework to build the analysis
from the ground up, and its credibility will show. Avoid forcing numbers. Be careful
not to work from the goals down to the supporting numbers. When working on the
analysis, many assumptions and variables may change numerous times throughout
the process, and these often will ripple to affect other parts of the analysis. Be sure to
check, double-check, and triple-check all the calculations. One error could shoot your
credibility in the entire analysis.
The checklist on the following page will help you build your business case for investing
in an integrated infrastructure.
31
Chapter 2
Building the Business Case for Integration
32
Executive Overview
M
ost midsize to large corporations and government organizations have a myriad
of computing platforms, business applications, databases, and other technolo-
gies distributed throughout their organizations. These systems were designed to meet
the needs of specific parts of the organization and were not intended to be integrated
with other systems. Business requirements have changed, however, and integration has
become a fundamental necessity for any business process that involves more than one
system or the coordination of work across different groups of employees.
Addressing a particular business need may involve several different integration re-
quirements. For example, implementing a supply chain management solution might
involve connecting electronically to suppliers and customers, managing the manual
processing of order exceptions, synchronizing order details among different internal
systems, and providing a single view of customer transactions and details across differ-
ent parts of the organization.
It is possible to break down almost all business integration requirementseven those
involving the most complex business processesinto a few basic integration scenarios
(or use cases, to use a software engineering term). These scenarios can be thought of
as building blocks for implementing more complex, integrated business solutions.
The basic integration scenarios are:
Synchronizing data among different databases and applications
Automating transactions among different internal systemscommonly re-
ferred to as enterprise application integration (EAI)
Integrating mainframe-based applications and data with distributed systems
Connecting businesses electronically and automating transactions with part-
ners, suppliers, and customersknown as business-to-business (B2B) integra-
tion
1.
2.
3.
4.
Common Integration Scenarios
3
33
Chapter 3
Common Integration Scenarios
Providing a single view of information that is distributed across different sys-
tems and databases
Extending access to systems over the Web and via other channels, such as mo-
bile devices and interactive voice response units
Repurposing IT assets by turning existing functions into components that are
then assembled into new composite applications
Automating and managing business processes and workflows
Using integration to optimize business performance
Just as the integration scenarios vary, so do the technologies available to implement
them. In this chapter, we discuss the integration scenarios and mention some of the
technologies. Then, in Chapter 4, we discuss the integration technologies in more de-
tail. Together, these two chapters provide a guide to choosing the right integration ap-
proaches and technologies for an organizations specific business requirements.
Data Synchronization
An organization usually has multiple systems and platforms, and similar information
is inevitably kept in many of them. Information about customers, orders, and products
can span customer management, order entry, fulfillment, warehousing, and billing sys-
tems. Keeping all this information synchronized is essential to the smooth execution of
many business processes.
Figure 3-1. Sharing data among multiple systems
5.
6.
7.
8.
9.
34
Chapter 3
Common Integration Scenarios
Many organizations synchronize data by rekeying information manually into systems.
However, this method has a high error rate, and fixing an error increases a transactions
cost by magnitudes. Automating data transfer provides direct cost benefits by reduc-
ing the personnel costs of both rekeying data and fixing errors. Some organizations
have found that error reduction alone delivers a sizable enough return on investment
(ROI) to justify an automated data integration solution.
Once an organization decides to automate data synchronization, it must choose which
approach to take: transferring flat files, performing database-level updates, or using
an application program interface (API) or application adapter. Because information re-
sides in disparate systems and platforms, data mapping and transformation must be
done so the information is delivered in the right format for the target platform. Data
synchronization patterns, from least to most complex, include one-to-one (synchroniz-
ing data between two systems), one-to-many (synchronizing data from one system
across two or more systems), and many-to-many (synchronizing data from multiple
systems across multiple systems). Data synchronization can also be one-way or bidirec-
tional. If it is bidirectional, a mechanism is needed to prevent an endless loop in which
an update to System A triggers an update to System B, which triggers another update
to System A, and so on.
Enterprise Application Integration
Whereas data synchronization addresses the need to duplicate data across different
systems, application integration addresses the need to link different applications in-
volved in the same business processfor example, an order management process that
spans customer relationship management (CRM) and enterprise resource planning
(ERP) packages. Mergers and acquisitions also create the need for application integra-
tion, for example, to integrate financial and HR systems.
35
Chapter 3
Common Integration Scenarios
Figure 3-2. Exchanging transactions among applications
Application integration is at the core of a variety of business problems. It can be a com-
ponent of B2B applications, composite applications, and business process automation
(as described later in the chapter).
Application integration can sometimes be achieved at the database levelfor exam-
ple, extracting an order from System As database and inserting it directly into Sys-
tem Bs database. In these cases, the programmer needs to understand the underlying
structure and meaning of the data and ensure that data consistency rules are adhered
to rigorously so the applications receive data in the format they require.
More typically, application integration occurs at the transaction level, such as during
the creation of an order. The transactionusually in the form of a message sent from
one system to anotheris typically processed through an API or an application adapter
that simplifies connectivity with the application. This level of integration enforces the
applications business logic. However, data translation and transformation are still nec-
essary to ensure that transactions are properly formatted for the receiving systems.
Critical components of application integration include transactional integrity, secu-
rity, the ability to trace what is happening, and exception-handling mechanisms. While
36
Chapter 3
Common Integration Scenarios
exceptions often must be processed manually, they should be managed as part of the
end-to-end automated and integrated process.
Application integration technologies include adapters, message brokers, EAI suites, en-
terprise service buses (ESBs), and application routers. Similar in concept to network
routers, application routers combine hardware and software to address simple integra-
tion needs.
Mainframe Integration
Mainframe integration can be part of various integration scenarios, including data syn-
chronization, application integration, B2B, composite applications, and multichannel
access. However, the unique intricacies of the mainframe environment demand ad-
ditional considerations.
In general, it is difficult to gain access to mainframe applications from the outside be-
cause they were not designed for this. Mainframe operating environments are tightly
controlled and do not lend themselves to external access beyond the originally pre-
scribed mechanisms, such as via terminals or reports.
Figure 3-3. Accessing mainframe systems and data
Mainframe access methods are broadly classified as invasive or noninvasive. Invasive
methods require changes to the mainframe code. They are difficult to implement but
37
Chapter 3
Common Integration Scenarios
may be worth the trouble when the integration needs to support a high transaction
volume. Noninvasive methods require few or small changes to the mainframe. In some
cases, no additional software is required. Options for noninvasive mainframe integra-
tion include the following:
Screen scraping, which replaces a human operator with a computer program
that interacts with the mainframe application through a terminal interface (for
example, 3270).
Report-based interfaces, which use the output from reports to trigger data-
oriented integration processes.
Direct data access via gateways to mainframe-resident databases such as DB2.
These products are sometimes referred to as SQL gateways.
Transaction-level integration to CICS or IMS. This can be noninvasive if the
mainframe application offers that interface, or minimally invasive if imple-
mented through a message-based interface.
In the long run, the best solution for mainframe integration is wrappering existing
mainframe transactionsthat is, using integration technology to noninvasively make
existing transactions accessible via different protocolsinto reusable Web services
that can be accessed from any source. Though it has an initial development cost, wrap-
pering enables greater reusability and flexibility.
Each approach to mainframe integration requires different technologies. The best ap-
proach depends on the characteristics of the mainframe application, the implementa-
tion time frame, and whether the integration is part of a tactical or a strategic imple-
mentation. Other factors may include the available skill sets, budget, and available tool
sets.
Market consolidation has left only a handful of specialized legacy integration vendors.
Many integration platform vendors partner with a specialist on OEM legacy integra-
tion technologies to offer mainframe integration capabilities as part of their broader
suites. Whether an organization is implementing a legacy extension solution as a tacti-
cal solution or building a legacy integration service layer as part of a Service Oriented
Architecture (SOA) strategy, it needs to understand how this capability fits into the
overall architecture.
4
4
4
4
38
Chapter 3
Common Integration Scenarios
B2B Integration
Automating transactions with partners, suppliers, and customers is commonly referred
to as business-to- business, or B2B, integration. Organizations require B2B integration
when they are attempting to streamline the supply chain, connect to an electronic
exchange, or comply with industry and regulatory mandates that require electronic
communications to government or business entities outside the organization.
Figure 3-4. Integrating business processes with trading partners
Processing electronic transactions with business partners can be 10 to 100 times less
expensive than processing the same transactions manually. Organizations typically
have used a value-added network (VAN) solution to implement electronic interchange
of purchase orders and other electronic documents, usually based on electronic data
interchange (EDI) formats and standards. However, VAN- and EDI-based solutions are
expensive to implement. This is due to the complexity of processing EDI and VAN
charges, which are based on the volume of data transacted. As a result, EDI is used
mostly by larger organizationsand by smaller organizations only when they need to
do business with larger organizations.
With the advent of Extensible Markup Language (XML) based integration technologies
that utilize the Internet and open standards rather than expensive proprietary technol-
ogies, alternatives have emerged. EDI is not going away, but EDI systems are evolving
39
Chapter 3
Common Integration Scenarios
to integrate with newer standards-based technologies. Together, the new integration
technologies are making it possible for many more organizations to participate in elec-
tronic commerce.
Integration with systems completely outside the organizations control pose some
challenging integration requirements. Additional security is necessary to authenticate
partners and ensure that the transactions are legal and cannot be repudiated. Orga-
nizations also need a way to manage service level agreements with each partner and
provide different connectivity options for small, midsize, and large organizations. B2B
integration also requires support for different data formats, including EDI, XML, and
custom flat files, as well as standard industry protocols, such as RosettaNet and the
Health Insurance Portability and Accountability Act (HIPAA).
B2B is also often associated with industry exchanges such Covisint, which started as an
automobile industry exchange and has expanded into healthcare, and the World Wide
Retail Exchange (WWRE), which was set up by 17 international retailers to automate
retail supply chain processes. These exchanges offer supply chain services that enable
suppliers and partners to conduct electronic business using the Internet rather than a
proprietary network as the transport layer. They allow all partners to integrate with the
exchange, rather than integrating point-to-point with all their partners and suppliers.
This saves considerable time and money.
Exchanges may also provide additional value-added services. For example, Covisint of-
fers a Web-based tool to coordinate and manage problem-solving processes between
customers and suppliers. WWRE offers a collaborative planning tool, a logistics tool,
and other facilities that enable on-line trading. Ariba, another B2B exchange, provides
buyers and suppliers with a number of services for conducting electronic business.
Today, B2B integration technologies are generally available as part of a broader integra-
tion platform or in a hosted environment through integration service providers, such
as GXS. Technologies for implementing B2B solutions include basic application capa-
bilities, partner management capabilities (which address the need to manage process-
ing rules and service levels on a partner-by-partner basis), and capabilities for handling
B2B-specific security requirements, such as firewall friendliness and the encryption of
documents exchanged over the Internet.
40
Chapter 3
Common Integration Scenarios
Case Study 3-1
Company: IT Corp.
Company Description
IT Corp. is a pseudonym for a Fortune 50 information technology company with more than
150,000 employees doing business in more than 170 countries. The company offers products
to consumers, small and midsize businesses, enterprises, and the public sector. It invests heav-
ily in R&D, producing more than 10 patents a day worldwide.
Business Challenges
IT Corp. has been dedicated to automating the computer industry supply chain and B2B trans-
actions. Traditionally, all supply chain transactions, from forecasts to invoices, were handled
via paper and email. This process was very inefficient and translated into high costs, increased
turnaround times, and data errors from manual processing. IT Corp. identified the supply chain
process as business-critical and wanted to create a Web-based channel that would enable
supply chain transactions between IT Corp.s customers and suppliers through a B2B Web site.
How Integration Addressed the Challenges
Using the business integration solution from webMethods, IT Corp. was able to link its Baan
ERP system with an internally developed Active Server Pages Web application. This enabled
the company to use all its existing business logic that resided in Baan for the new Web appli-
cation. With the Web-based supply chain system, business partners can now conduct all trans-
actions on the B2B Web site. Customers can place new orders, request changes to existing or-
ders, view and print order acknowledgments and shipping details, and submit order forecasts
to IT Corp. to streamline the procurement process. Suppliers can easily view and acknowledge
orders, provide updated shipment details, and access procurement forecasts to manage sup-
plies. The new system processes between 10,000 and 15,000 transactions each month.
Bottom-Line Business Benefits
Delivered in just three months, the Web-based B2B portal has become an integral part of IT
Corp.s supply chain system and supports a growing percentage of the companys transactions.
The automated process has streamlined and simplified the purchasing process for the com-
panys suppliers and customers. By completing simple online transactions, customers receive
immediate responses and acknowledgments of process completion, and this has increased
customer satisfaction. In addition, the automated system has eliminated time-consuming and
error-prone manual processes. The real-time nature of data flows across the system provides
accurate information at all times. The new system has effectively saved thousands of man-
hours of work for IT Corp. and its business partners.
41
Chapter 3
Common Integration Scenarios
Because most organizations do business with many types of partners, they need differ-
ent types of on-ramps, or methods of connecting to partners and suppliers. The larg-
est partners may have full-blown B2B implementations, including full integration with
their back-end systems. Some midsize partners may just need to send and receive XML
messages. And smaller partners may require browser-based connectivity where they
manually submit transactions or check on status. Ultimately, the goal is to make it fast
and easy to connect, because the ROI for B2B implementations is directly related to the
percentage of partners participating.
Single View of Information
With the proliferation of applications and databases in an organization, information
about customers, suppliers, employees, products, orders, and other business entities
becomes spread out, and it is difficult to get a single, consistent view. Integration soft-
ware can help organizations aggregate information from various systems to create a
single view of any business entity. This capability is especially important for business
solutions used in situations such as mergers and acquisitions, CRM, or integrated risk
management.
Figure 3-5. Combining information into a single view
42
Chapter 3
Common Integration Scenarios
One way to gain a single view is to physically aggregate data into one place. Many
organizations have created data warehouses and data marts to accomplish this. How-
ever, data marts and warehouses are generally updated at predefined intervals, such
as nightly or weekly, so they may lack the most current information. An alternative
method is the distributed query, which involves keeping the information in the source
systems and extracting the information as needed into a combined view, as if all the
information were coming from a single source. This ensures that the information pre-
sented is up-to-date and the most current available. The tradeoff with a distributed
query is that it typically takes longer to get a response than it does when making a
request to a single database.
An important dimension of single-view integration is the need to understand both the
underlying data structures of the various source systems and the semantic meaning of
the data in each system so they can be reconciled into the single view (whether in a
data warehouse or in real-time). It is necessary to determine, for example, that the Cus-
tomer field in System A is the same as Name in System B, CustName in System C, and
Client in System D, and to map the transformation of data as it is extracted from the
sources. A major ambition for data integration is to capture the semantic meaning of
information within systems in metadata, which allows systems to be integrated more
easily without having to create each mapping manually.
A number of new solutions are coming to market that enable a real-time single view
of information. These include enterprise information integration (EII) solutions, which
have proprietary technologies and algorithms for aggregating information in real-
time, such as distributed query optimization, indexing, and caching. Some solutions
include semantic integration capabilities. For physical-aggregation-style approaches,
traditional data integration technologies can be used to source data fromand popu-
late data inthe databases involved.
43
Chapter 3
Common Integration Scenarios
Case Study 3-2
Company: FoodCo
Company Description
FoodCo is a pseudonym for one of the leading food and beverage companies in the United
States. It is the largest processor and distributor of milk and other dairy products, and oper-
ates more than 120 plants in the United States and Spain.
Business Challenges
Over the years, FoodCo had acquired several new businesses, which operate as independent
business units. Each acquisition brought more SKUs or UPC codes, and managing them had
become increasingly challenging. Data for identical products was inconsistent among busi-
ness units, and finding and validating information that customers needed took a long time. In
addition, maintaining product information was a complex and error-prone process, especially
when items had multiple levels of descriptions, such as formulas for ice cream flavors. FoodCo
realized that it needed to create an enterprise-wide product data management process to
ensure data consistency across business units and guarantee the delivery of information to the
right place at the right time.
How Integration Addressed the Challenges
Using the webMethods integration solution, FoodCo created a single master-item system that
aggregates data from multiple back-end systems and routes it to users for action. The new
system enables more than 100 employees across 32 business units to create, edit, validate,
and manage product data. The system also generates alerts to resolve exceptions that could
lead to data inconsistency.
Customers now see FoodCo as a single vendor with a consistent product catalog. They can
make purchases in a timely manner because they receive notification beforehand and can
react more efficiently to what the company sends them.
Bottom-Line Business Benefits
The new master-item system enables FoodCo to manage more than 35,000 SKUs in nine
categories and to add more than 200 new items each year. For its 360,000 customers and
partners, the system has reduced the time for new product introductions from five days to
one. Because the enterprise integration platform synchronizes the data within the existing ERP
systems, FoodCo can leverage its investment in internally developed systems and meet cus-
tomer mandates on synchronized items. The webMethods platform has also enabled FoodCo
to implement a complete standards-based EDIINT/AS2 solution between VAN and non-VAN
customers, resulting in a five-figure savings on annual VAN charges.
44
Chapter 3
Common Integration Scenarios
Multichannel Access
The past few years have seen a proliferation of devices through which end users and
consumers can access computing services. No longer tethered to an application run-
ning on a desktop computeror, before that, a mainframe terminalpeople can now
use Web browsers, personal digital assistants (PDAs), mobile phones, interactive voice
response units, and other special-purpose devices to receive and send information. As
organizations start creating reusable business services, they inevitably want to make
these services available through different channels.
The expansion of access channels represents an opportunity for organizations to bet-
ter serve their employees, customers, suppliers, and partners anytime and anywhere,
but it also presents a significant IT challenge. The associated technologies are expand-
ing and changing rapidly, and organizations will increasingly need to provide services
across a variety of devices based on a variety of technology environments.
Rather than altering existing applications to allow for each type of access channel, or
developing solutions that are hard-wired to a specific type of device, several emerging
technologies and standards initiatives allow multichannel access to different back-end
systems.
Figure 3-6. Using different end-user channels
to access back-end systems
45
Chapter 3
Common Integration Scenarios
Case Study 3-3
Company: Comco, Inc.
Company Description
Comco, Inc. is a pseudonym for a U.S.-based Fortune 100 global communications company
that provides mobility products and solutions for the home, automobile, and workplace. The
companys 2004 sales exceeded $30 billion.
Business Challenges
Comcos mobile device business unit was losing significant margin because of duty fees for
phones shipped from its U.S. plants to Latin America. The company recognized that divesting
the manufacturing of these phones to a regional manufacturer would reduce its duty fee costs
in the Latin America market by several million dollars each year. Comcos major challenge was
to define an automated process that would transfer the manufacturing and distribution of
mobile phones to the third-party contractor without reducing on-time shipments to Comcos
customers. Another challenge was transforming Comcos existing ERP system into two sepa-
rate but seamless systems, one located at the new contractors plant. This would involve a
cutover process with minimal interruptions for each company.
How Integration Addressed the Challenges
Comco had made a formal commitment to RosettaNet, an industry standard for B2B com-
merce. It wanted to build on the RosettaNet work and create an enterprise integration solu-
tion that would provide touchless integration from Comcos back-end Oracle ERP system to
the third-party contractors back-end SAP system. Comco implemented a webMethods-based
solution that enabled the new system to automatically exchange a variety of information
about engineering, purchase orders, and other product information. The system was also
enhanced with business activity monitoring (BAM) capabilities that send alerts to business
managers when process exceptions occur, allowing them to react quickly and proactively.
Bottom-Line Business Benefits
The new system has enabled Comco to relocate its manufacturing to Latin America and avoid
several million dollars in duties annually. The company has also reduced its operating costs by
close to $1 million each year through the elimination of manual processes, effectively reducing
the number of people required to support the business while maintaining on-time shipments
to customers. To date, the touchless system has enabled more than 1.2 million mobile phones
to be shipped to Latin American customers with no direct contact by Comco. In addition, order
tracking and other information is provided in real-time, giving the company added bandwidth
to focus on new business opportunities. This is a significant competitive advantage for Comco.
46
Chapter 3
Common Integration Scenarios
A midtier server is the least invasive and most flexible way to provide interface ser-
vices to multiple types of front-end devices. The midtier server renders the interface to
fit the display characteristics of each device. This way, the back end does not need to
change the way it sends the information for each type of device.
For accessing back-end systems, Web services are a good choice because they enable
access from a broad range of clients-side systems, including Web servers, Java, C#, and
mobile devices. Additionally, the use of XML and style sheets enables information to be
displayed in different formats for different devices without requiring changes to the
underlying data.
Few organizations will be able to ignore the demands for multichannel access from
multiple constituencies, and for many it will soon become a requirement. Organiza-
tions therefore need to create a strategy and an architecture for multichannel access
and integration.
Composite Applications
Organizations seeking to rapidly implement new business solutions soon realize that
they cannot pursue a rip-and-replace strategy. Starting fresh with each new system
simply costs too much and is too disruptive to the organization. Unfortunately, many
applications hinder rather than enable business change. Their rules, processes, data
definitions, and user interfaces are embedded in compiled code that requires program-
ming skills and detailed background knowledge to modify. To meet business needs,
organizations must leverage the systems they have in place but gain flexibility in how
the systems can be repurposed. Composite applications are the answer.
Composite applications are assembled (or composed) from existing components rath-
er than developed from scratch. They also incorporate functionality from a variety of
underlying systems.
This approach to building new systems aligns closely with Service Oriented Architec-
ture. SOA is an architectural approach to creating reusable building-block components
47
Chapter 3
Common Integration Scenarios
as services that interoperate through standardized Web services interfaces and that
can be combined quickly and easily to address new business process needs. A compos-
ite application consists of functionality drawn from several sources within an SOA. The
components may be individual Web services, selected functions from other applica-
tions, or entire systems whose business functions have been enabled as Web services
(often legacy systems).
Figure 3-7. Composing new applications from
existing underlying business services
An example of a composite application is a self-service portal for consumers or busi-
ness partners. Here, a number of existing processes supported by IT services might be
combined within the portal to provide the end user with one-stop access to a range
of the organizations products and services. Without an SOA approach, this type of
composition would require considerable application integration, programming, and
testing. With an SOA, however, the composite application can be assembled easily,
48
Chapter 3
Common Integration Scenarios
regardless of the location or the technology of the different services being drawn to-
gether. This allows IT to respond faster to business demands, keeps costs to a minimum,
and powers imaginative new ways to address end users needs more effectively.
One caveat must be mentioned here. Whereas a monolithic application typically pro-
vides tools to monitor status (for example, to view the status of an order), a composite
application offers no such built-in monitors because the status crosses multiple ser-
vices on potentially different platforms. Therefore, for monitoring and other functions
such as policy enforcement, composite application implementations need an infra-
structure that exists outside the services themselves.
The technologies that enable composite applications include traditional application
development platforms, integration suites, Web services, and Web service repositories
that enforce governance policies, provide security, and enable reuse. The services from
which composite applications are composed can be created by wrappering existing
application functions or by building new ones from scratch. As such, a tool set is need-
ed to catalog, discover, and orchestrate services into a new application, and to deploy
and manage the result. Additionally, a run-time deployment infrastructure is needed to
connect all the components in a scalable and reliable way.
Business Process Management
As integration technologies and solutions have matured, the scope and complexity of
the problems that organizations are looking to solve have increased. Beyond ensuring
data consistency, linking a few applications together, or exchanging documents elec-
tronically with a trading partner, organizations are increasingly looking to integration
software to streamline entire business processes.
Business processes are the lifeblood of an organization. Whether related to customer
service, product development, or service delivery, an organizations business processes
define its ability to compete successfully in the market. Process-level integration offers
the richest business benefits. Organizations can increase competitiveness through dif-
ferentiated business processes, and IT can better keep pace with changing business
requirements. So, while eliminating manual data entry and automating transactions
can yield direct and measurable cost benefits, automating and optimizing business
49
Chapter 3
Common Integration Scenarios
processes holds the greatest potential for reducing labor costs and cycle times; im-
proving employee, customer, and partner satisfaction; and increasing business agility
Figure 3-8. Coordinating end-to-end business processes
across systems and people.
By nature, business processes are often complex and disjointed. Different organization-
al entities may be responsible for different parts of the process, with no single entity
responsible for the end-to-end process. In addition, business processes often include
both automated and human processes. Some parts of the process may be performed
automatically by IT systems, whereas other parts may involve people performing spe-
cific tasks. In fact, typical enterprise business processes run the gamut from fully au-
tomated to semi-automated to entirely manual. Although organizations may want to
automate as much of the process as possible, people remain an important part of the
equation. Individuals are needed to assess exceptions, apply judgment and experience,
and handle other situations requiring context that is not easily encoded in a system.
The term business process management (BPM) generally refers to the set of activities
that organizations perform to either manage or optimize their business processes and
adapt them to new organizational needs. Because these activities are usually aided by
software tools, BPM also refers to the category of integration software that provides
these capabilities.
50
Chapter 3
Common Integration Scenarios
Since business processes are typically a combination of automated and manual tasks,
BPM solutions should support both types of processes. Human processes require a
user interface. Typical human processes involve delegation and organizational consid-
erations, such as the different roles of managers and employees. For example, manag-
ers need the ability to assign task to employees with certain skill sets, manage their
employees workloads, balance the workloads when assigning or reassigning tasks,
and provide management oversight and intervention for manual activities.
BPM solutions typically provide graphical tools for modeling business processes, a plat-
form for automating and executing these processes, and facilities to monitor them. They
may also provide simulation capabilities to test and optimize business processes.
Business Performance Optimization
Henry David Thoreau said, In the long run, we only hit what we aim at. When organiza-
tions have tools that deliver real-time visibility intoand performance measurements
ofbusiness processes, they can focus not only on what goods and services they are
delivering but also on how they are delivering those goods and services. This focus is
necessary for business performance optimization.
Business performance optimization is a combination of methodology, best practices,
and technology infrastructure. There are two levels of business optimization. As dis-
cussed above, business process management solutions enable process-level optimi-
zationin essence, streamlining the execution of business processes. Business per-
formance management solutions monitor relevant business events, which may cross
multiple processes, and relate them to key performance indicators (KPIs).
Both levels of business optimization are required. In fact, it can be argued that business
optimization should be performed at all levels of the organization, from the operational
IT level, through line-of-business managers, up to senior executives. All require visibility
into the KPIs for which they are responsible. Alignment across all organizational levels
is required for overall business optimization. Otherwise, an organization risks optimiz-
ing one process while suboptimizing at an enterprise level.
51
Chapter 3
Common Integration Scenarios
Business activity monitoring (BAM) is an emerging technology for enabling business
performance optimization. In 2002, Gartner coined the term BAM and defined it as
the concept of providing real-time access to critical business performance indicators,
along with the supporting information to improve the speed and effectiveness of busi-
ness operations. Whereas BPM solutions provide visibility across individual business
processes (or process-level management), BAM solutions allow organizations to corre-
late events from multiple processes and relate them to measures that are meaningful
to the enterprise. In doing so, BAM provides a big picture of business performance
and helps align business strategy with operational execution.
BAM solutions provide business dashboards that correlate real-time business events
with KPIs. Some BAM solutions include predictive capabilities and can identify emerg-
ing patterns that may influence a KPI, giving managers the information necessary to
proactively handle emerging problems rather than simply react to erupting crises.
Figure 3-9. Improving business performance
with BAM technology
52
Chapter 3
Common Integration Scenarios
BAM is often associated with business intelligence (BI) technology. Consider them
complementary sides of a coin, with BAM providing the real-time operational view into
business performance and BI the view for historical analytical purposes. Some BAM
vendors offer the ability to integrate with and feed into existing data warehouses, al-
lowing organizations to use their BI tool of preference.
Best Practices
This chapter illustrates how integration can be applied to a broad range of business
requirements. Chapter 4 provides more detail on the underlying technologies used to
implement the scenarios described in this chapter.
It is important to understand that midsize and large organizations have a variety of
business requirements and therefore require a variety of integration technologies. Be-
low are some best practices to increase the odds that an organizations integration
initiatives will succeed in the long run.
PIan strategicaIIy, impIement tacticaIIy. Each of the integration scenarios presented
in this chapter could be approached as a tactical implementation or as a necessary
technical task within an overall strategic implementation. While such an approach
might yield short-term benefits for low-hanging fruit, in the long run it will increase
redundancy, software license costs, skills training, and management and maintenance
costs. In contrast, an enterprise architecture approach to integration will enable long-
term business agility. It will make different integration technologies available to differ-
ent projects as needed, ensure that the technologies are integrated with one another,
and decrease the time and cost to implement new solutions.
Create an Integration Competency Center (ICC). As an organization strives to de-
velop core competencies for managing integration, an ICC will help it more efficiently
build and maintain the complex integrations that its business demands. The ICC is re-
sponsible for the development and deployment standards that will help facilitate reuse
and interoperability as well support the implementation, maintenance, and continual
evolution of the organizations integration platform. (See Chapter 6 for a description of
the ICCs function and roles.)
53
Chapter 3
Common Integration Scenarios
Practice continuous improvement. Business agility is a journey, not a destination. Or-
ganizations need to evolve their infrastructures, business services, and processes to
optimize the business at all levels. Technology is just part of the solution. Real-time
visibility into processes and KPIs is only useful if someone is paying attention and prac-
ticing continuous improvement based on the enhanced information. An organizations
success depends largely on what it does with the technology. Organizations need to
build a culture of continuous improvement and business optimization. This requires
that executives, line-of-business managers, and business operatives are all aligned
along the same business goals, and that a reward structure is in place to promote the
desired outcomes. The good news is that integration technology can also help solve
this problem through management and operational dashboards.
54
Executive Overview
A
s mentioned in Chapter 3, different business scenarios warrant the use of different
integration technologies and infrastructure services. Therefore, organizations will
inevitably acquire a collection of technologies to meet their diverse business needs. To
maximize agility and reuse, organizations need to understand how the range of inte-
gration solutions available today relate in the overall enterprise architecture.
The ebizQ Integration Roadmap can help organizations match their business require-
ments with the appropriate integration technologies. Becoming familiar with the
Roadmap is a good way to understand both the integration landscape and the types
of business problems addressed by specific technologies.
Application Integration Services
Application integration services offer the basic features and capabilities required to
integrate and connect applications reliably and securely. These services include:
Connectivity to the applications. Connectivity is generally achieved using
adapters, which enable a standard application connectivity approach and
eliminate the need to program against proprietary application program inter-
faces (APIs). If Web services interfaces are available, application connectivity
may be achieved without an adapter. Application connectivity may also be
achieved at the database level using a database adapter or a direct SQL inter-
face.
Data transport and routing capabilities to move information reliably and se-
curely between systems. Application integration solutions typically use a mes-
sage-based approach to moving data and support a variety of topologies that
link systems together in a scalable and flexible manner.
Data transformation and mapping capabilities. These enable data to be con-
verted to and from the required target representations.
4
4
4
Guide to Integration Technologies
4
55
Chapter 4
Guide to Integration Technologies
Infrastructure services to implement the application integration scenarios de-
scribed in the previous chapter. These include support for transactions, appli-
cation-level security, and exception management services.
Figure 4-1. ebizQ Integration Roadmap
The core application integration capabilities often play a role in other integration ser-
vices, such as information integration, interface integration, composite applications,
process integration, and business optimization with business activity monitoring
(BAM). Without the underlying application integration services, these other integration
services require point-to-point connectivity with back-end applications, which limits
agility and reuse.
Technologies that provide application integration services include message brokers,
integration servers, and enterprise service buses (ESBs). In addition, connecting with
mainframe systems and integrating the systems of different organizations in a busi-
ness-to-business (B2B) scenario have special considerations and, consequently, require
additional technology solutions.
4
56
Chapter 4
Guide to Integration Technologies
message BrOkers
The original application integration technology, message brokers provide the basic
connectivity needed for application integration, including messaging, data transla-
tion and transformation, and intelligent message parsing and routing. Message broker
products support different styles of messagingsuch as request/reply for sending a
message to an application and receiving a reply, and publish/subscribe for broadcast-
ing information and events to a number of applications. An important function of the
message broker is to provide a secure and reliable communications layerfor exam-
ple, to ensure the guaranteed delivery of a message from one application to another.
Message brokers usually have a hub-and-spoke architecture, with all processing done
at the central broker. More advanced vendor solutions offer distributed processing op-
tions and multiple hubs to improve scalability and availability.
InTegraTIOn servers
As application integration requirements have become more complex, the basic mes-
sage broker-centric architecture has evolved to provide more functionality at the in-
tegration end points, such as supporting more sophisticated application processing
rules, handling B2B standards like RosettaNet, and performing protocol translation (for
example, layering a Web services interface around an existing application interface).
These additional capabilities are typically combined into an integration server. Inte-
gration servers can be used in combination with message brokers to create a scalable
integration backbone, with multiple brokers connecting multiple integration servers.
They can also be used alone for smaller point-to-point integration projects.
InTegraTIOn plaTfOrms
Since introducing their products in the mid-1990s, enterprise application integration
(EAI) vendors have evolved their message brokers, integration servers, and adapter li-
braries into integration platforms, which include additional capabilities and services
to address different integration scenarios and provide more direct business value.
These platforms, also called integration suites, generally include full B2B capabilities
57
Chapter 4
Guide to Integration Technologies
and some level of business process management (BPM) and workflow capability. They
may also include business activity monitoring features, a portal, mainframe integration
capabilities, and other combinations of the integration technologies. These additional
capabilities are discussed later in this chapter.
esBs
The enterprise service bus is an emerging middleware category that is typically viewed
as a key enabling technology for Service Oriented Architecture (SOA). Just as previous
generations of distributed systems depended on middleware such as RPC, CORBA, and
transaction monitors, SOA relies on ESBs to help bind the services together.
ESBs serve as intermediaries between services providers and service consumers to
avoid point-to-point connections. In this role, ESBs can fulfill requirements such as in-
telligent routing, transformation, load balancing, and security while also coordinating
the flow of service calls. Conceptually, ESBs provide much the same functionality in an
SOA as an integration platform does in an application integration scenario. The biggest
difference is that ESBs are generally based on Web services standards.
Some vendors originally positioned ESBs as simpler alternatives to full-blown integra-
tion platforms. In reality, the lines have blurred as ESB vendors incorporate functional-
ity similar to traditional integration platforms and suites.
adapTers
In a typical integration scenario, messages are routed between the various applica-
tions being integrated. Once a message arrives at an application, it must be processed,
usually by making updates or changes to the application or its database. The options
for accessing the application include writing a program that uses the API, using a pre-
built adapter, or using a Web services interface.
Many commercial applications have APIs, which enforce the necessary business rules,
transaction logic, and data checks for programs that interface with the application. Ac-
cessing the application via the API is thus preferred over integrating directly at the
58
Chapter 4
Guide to Integration Technologies
database level. The downside to APIs is that they differ for every application, so the
developer needs to know how to code to each applications unique APIs. This increases
the time and cost for integrating systems and decreases agility.
Adapters are a desirable alternative to API-level programming. They simplify the inte-
gration effort by eliminating the need for application-specific interface development.
Adapters provide a common set of function calls for different back-end applications.
On the one side, an adapter exposes a standard interface to the developer; on the oth-
er side, the adapter deals internally with the task of translating these interface calls
to the specific APIs of a supported application. The developer needs only learn how
to code the standard adapter interface rather than each applications interface. Some
vendors reduce the development effort further by offering graphical tools for config-
uring adapters.
There are two types of adapters. Intelligent adapters can perform data translation and
transformation at the source or target, thereby distributing the processing load and
increasing the solutions overall scalability. Standard (non-intelligent) adapters provide
only basic connectivity services, thus requiring a message broker or integration server
for translation and transformation.
Off-the-shelf adapters are not always available for every application that an organi-
zation might want to integrate. So vendors frequently provide a development kit or
library for building custom, one-off adapters that interface with the application while
providing the niceties of a commercial adapter, such as graphical configuration or
built-in logging and error handling.
WeB servICes
Web services are an attractive integration alternative because they provide a standard
interface protocol that eliminates the need for adapters. However, Web services only
replace adapters, not the rest of the application integration services. For example, they
do not perform functions such as data translation and transformation, which remain
an important feature of the overall integration solution.
59
Chapter 4
Guide to Integration Technologies
B2B InTegraTIOn
Additional requirements exist when integrating with applications beyond the firewall.
These requirements include:
Support for a variety of electronic commerce standards and formats,
including Electronic Data Interchange (EDI) and newer XML-based
standards such as RosettaNet, as well as support for custom document and
flat-file formats.
Support for connectivity to proprietary communications networks and over
the Internet using protocols such as FTP, HTTP, and Web services.
More stringent security than what is typical for internal application integra-
tion. This includes the need to manage data confidentiality and integrity and
the identity of the parties involved.
Management of configuration settings and service levels on a partner-by-
partner basis.
B2B integration solutions usually include a variety of connectivity optionssometimes
called on-rampsfor different types of partners. The challenge of B2B commerce is to
provide a range of access technologies to meet the needs of all partners and suppli-
ers. For larger partners, it might make sense to fully automate the B2B transactions,
including integrating with the back-end systems on both sides of the firewall. Smaller
partners with minimal IT resources require simpler solutions, such as partner portals
for inputting information or just sending XML messages. Tools that enable organiza-
tions to connect faster, such as integration and process templates, accelerate the time
to value.
A major evolution of B2B integration is in the area of EDI. Electronic Data Interchange
encompasses a set of standard protocols for conducting structured interorganization
exchanges, such as making purchases or initiating loan requests. EDI transactions are
typically conducted through a value-added network (VAN) provider. These VANs are
essentially private networks for conducting B2B transactions.
4
4
4
4
60
Chapter 4
Guide to Integration Technologies
The explosive use of the Internet has led to the creation of a new standardEDIINT
that enables EDI transactions to be integrated with Internet transactions using XML.
EDIINT has several variations, which are based on the transfer protocol used. EDIINT ASI
uses SMTP, while EDIINT AS2 uses HTTP and HTTPS for securely exchanging EDI files
over the Internet. Some integration solutions on the market today combine EDI and
XML standards support in the same framework and tool set, allowing a single solution
to be used for all B2B-related integration requirements.
B2B solutions also demand differentand in many cases greaterlevels of security
than does internal application integration. Authentication services are required to con-
firm the users or organizations identity. Authorization services are required to specify
a user groups levels of access to various applications and services. And nonrepudiation
services are required to ensure the validity and enforceability of electronic contracts.
The type of nonrepudiation needed depends on the type of application. Nonrepudi-
ated communication sessions may suffice for simple data exchanges, but nonrepudiat-
ed transactions are generally required when the solution includes more complex B2B
transaction processing.
B2B integration exchanges and service providers have become viable alternatives for
essentially outsourcing the B2B solution. Each partner needs only connect with the ex-
change rather than manage the details of connecting with every partner. This reduces an
organizations burden when connecting to multiple partners with different technology
infrastructures.
maInframe InTegraTIOn
Mainframe systems usually do not have exposed APIs and therefore present unique
application integration challenges. There are several ways to integrate with these
legacy systems. Options include screen scraping or report scraping, integrating with
the legacy database, integrating at the transaction level through specialized gateways
(for example, for IBM CICS or IMS transactions), wrappering legacy code as a service
with a standardized interface, such as a Web service, or front-ending (that is, creating
a message-oriented interface to an application), which requires some changes to the
61
Chapter 4
Guide to Integration Technologies
application. Specialized legacy integration solutions are often used in combination
with other integration software to connect the mainframe systems with applications
running on distributed platforms.
Information Integration Services
Information integration services consolidate and integrate structured and unstruc-
tured information from multiple sources, and manage it at an enterprise level to pro-
mote data consistency. The value of the information in an enterprise system is in direct
proportion to its quality and consistency with other systems. When data is managed as
an important enterprise resource, it provides greater value.
The key to managing distributed information effectively is metadata, which is informa-
tion about data represented independently from the underlying systems. Metadata
plays an important role in enabling interoperability among systems because it allows
data to be understood and exchanged consistently across the systems. Enterprise
metadata repositories that provide real-time data services enable reuse and agility.
The next leap forward in data management services and integration productivity will
come from semantic integration. Semantic integration captures the meaning and con-
text of data within source systems in a metadata repository. Every integration project
requires an understanding of the underlying information in the systems. It is crucial to
know, for example, that CustName in one system is the same as Company in another
system. Generally, this knowledge is put into proprietary mapping tools. By captur-
ing it as semantic metadata, organizations can maximize reuse and ultimately enable
systems to automate the rules for translation and transformation. This also increases
return on investment (ROI) because the effort and cost of discovering the semantic
meaning of data within systems can be more readily reused across multiple projects.
Enterprise information integration (EII), enterprise content management (ECM), and
metadata repositories are the newer technologies that focus on information integra-
tion. Additionally, organizations will continue to use traditional data integration ap-
proachessuch as data warehouses and extract, transform, and load (ETL) toolsto
deliver the full set of information integration services.
62
Chapter 4
Guide to Integration Technologies
eTl
Extract, transfer, and load technology was developed for data warehousing implemen-
tations. ETL tools operated on large data sets in batch mode, extracting the informa-
tion from systems on a nightly or weekly basis, cleansing it, transforming it into the
desired target format, and then loading it into the data warehouse or data mart. ETL
technology is still used today for applications where it is preferable to physically move
the information to create a unified view and where real-time updates are not neces-
sary, such as for historical analytics. However, the newer information integration tech-
nologies discussed below offer the option of leaving the data in the source systems
and retrieving it in real-time as needed.
eII
An emerging integration market sector, enterprise information integration provides an
aggregated view of data. EII solutions can provide the equivalent of a virtual data ware-
house or midtier data store for real-time reporting or fast implementation of a portal
solution. EII tools have technologies for distributed query optimization, indexing, and
caching that enable real-time information access and aggregation. Graphical views
make it easier for programmers and business managers to access the information they
need without knowing the underlying data structures of the source systems. Whereas
data warehousing requires that the information be moved to a central database, EII
solutions access the data from the source systems themselves. Emerging EII solutions
are also incorporating semantic integration capabilities.
eCm
Enterprise content management provides integrated access to and management of
unstructured data, such as documents and images. ECM solutions can provide work-
flow and process management as well as integration capabilities. ECM services are re-
quired when the business process is primarily focused on documents being passed
around the organization, or when unstructured content needs to be part of an overall
business process integration solution.
63
Chapter 4
Guide to Integration Technologies
Interface Integration Services
Interface integration services provide access from a variety of front-end devices to a
variety of back-end systems. The technologies that provide interface integration ser-
vices include portals (for delivering the user interface via a browser) and mobile inte-
gration gateways (for enabling the use of mobile devices).
pOrTals
Portals provide an integrated user interface to multiple back-end systems. Many EAI
solutions focus on automating transactions across systems. But a portal solution inte-
grates information into a browser for a particular purpose or set of users, providing a
single touch point for the delivery of application services.
Various types of portals exist. Somesuch as enterprise information portals, which pro-
vide information on corporate policies or newsare informational only. That is, they
aggregate information and functionality from multiple systems, but the information
flows only one way: to the portal. Transactional portals provide greater interoperability;
for example, enabling customers to place and track orders, or allowing employees to
enter expenses or manage their 401ks. There are also management portals for tracking
business processes or key performance indicators (KPIs), and operational management
portals for monitoring system-level performance.
Portal technology often includes an environment for developing the portal applica-
tion. This application can be composed of multiple portlets, which are pluggable user
interface components. For example, there may be a log-in portlet that manages log-ins
to multiple back-end services. Portals may also include personalization features that al-
low users to customize the look and feel of the portal applicationfor example, choos-
ing what types of information to display, such as local weather or a customized stock
ticker. Although portal technology provides a way to develop an integrated user inter-
face, back-end integration is still required. This can be done point-to-point or using the
application integration technologies discussed above.
64
Chapter 4
Guide to Integration Technologies
Several important standards address portals. Web Services for Remote Portlets (WSRP)
defines a set of standard interfaces for creating portlets that can be accessed remotely
using Web services protocols. This allows applications to consume portlet components
without having to write unique code for interacting with each component. And JSR
168 is a Java API for portlets that enables content sources and application front ends to
be aggregated. It also addresses how security and personalization are handled.
mOBIle InTegraTIOn gaTeWays
Mobile devicesincluding cell phones, PDAs, and specialized mobile units such as
package tracking systemsall have different screen sizes, resolutions, and interface
requirements. These mobile devices are proliferating rapidly. Today, people commonly
use their cell phones to receive email, surf the Web, and conduct other aspects of busi-
ness.
IT departments are coming under increasing pressure to provide enterprise application
services to a variety of mobile devices. Mobile applications are likely to require back-
end integration to present information from multiple sources. In addition, because
mobile devices operate on inherently unreliable and insecure networks, management
and security need to be part of the integration solution. And mobile applications of-
ten must accommodate occasionally connected users, who need to work offline and
then synchronize with source systems and reconcile the offline transactions across the
systems.
Mobile integration solutions usually include a specialized server that provides the
transformation services for each target device, along with security. This server inte-
grates with back-end systems either directly (point-to-point) or through an integra-
tion platform. The mobile integration server typically uses message queues and logs
to guarantee that messages will be delivered once, and only once, even in the event of
a system failure. To support occasional users, it controls message delivery by organiz-
ing messages into transactional groups, and determines when messages are delivered
based on the type of network connection. To address the insecurity of mobile commu-
nications, the ability to encrypt data is important.
65
Chapter 4
Guide to Integration Technologies
Composite Application Services
The term composite application refers to an application assembled from components
that may originate from various underlying systems. Composite applications enable
new business solutions to be created by flexibly combining new and existing appli-
cation functionality. This allows organizations to deliver systems faster than building
them from scratch. The benefits of a composite application approach are maximized
when organizations create reusable business serviceswhich then serve as building
blocks for a composite applicationas part of a Service Oriented Architecture. As a
result, composite applications are typically associated with SOA and Web services.
Composite application services are the technical infrastructure services that support
the development, deployment, and management of composite applications. These in-
clude the ability to:
Develop new services (typically based on Web services standards)
Create service abstractions by combining several low-level services into a
more useful, reusable high-level business service
Design and model the behavior of the composite application, including how
information and messages flow across services, and how services need to be
orchestrated in support of a business process
Discover services or browse from a registry of available services
Manage policies and service level agreements to engender trust and enable
the reuse of services
Deploy the resulting solution into a managed run-time environment
The technologies most commonly associated with composite applications include
Web services, service orchestration capabilities, and service registries for governance.
Software vendors are introducing composite application platforms to offer a collection
of these technologies and services in an integrated suite.
4
4
4
4
4
4
66
Chapter 4
Guide to Integration Technologies
WeB servICes
Web services and their related standards have emerged as the de facto building blocks
for composite applications. Web services define a standardized interface (Web Services
Description Language, or WSDL), a standardized communication protocol (Simple Ob-
ject Access Protocol, or SOAP), a standardized repository for registering and discover-
ing Web services (Universal Description, Discovery, and Integration, known as UDDI),
and standardized message encoding using XML. These standards enable a Web service
to reside anywhere and be accessed from everywhere, making Web services well suit-
ed to the role of providing aggregated functionality within a composite application
and being the standard for interapplication interfaces. As a result, many organizations
today base their application development and integration efforts on Web services.
OrChesTraTIOn
To create a composite application from a set of Web services, it is necessary to define
the flow of control across services. Doing this in programming code reduces the flex-
ibility to change the application, so an externalized approach to service orchestration
is preferable. This latter approach basically allows developers to modify the way ser-
vices are linked together without changing the services themselves. Business Process
Execution Language (BPEL), a vendor-led initiative, is gaining mind share and market
share as the orchestration standard. A number of software vendors now support BPEL,
making it possible to share process definitions more easily among tools.
servICe regIsTrIes fOr gOvernanCe
UDDI is the registry standard for dynamically discovering and invoking Web servic-
es. UDDI was originally created to manage the cataloging of public Web services, but
registries are also gaining ground within the enterprise. Organizations that embrace
SOA will experience a proliferation of servicesand versions of services. To reap the
benefits of reusability, organizations must manage these services effectively through-
out their life cycle. Beyond basic UDDI support, commercial registry products typically
incorporate some level of Web services management capability to make the registry
more useful within the enterprise.
67
Chapter 4
Guide to Integration Technologies
In addition, some vendors use the registry as the basis for policy and governance man-
agement in their solutions so that policiessuch as service level agreementscan be
enforced at run-time. This approach allows the policies to be enforced across all ap-
plications that utilize the service, rather than having each application implement the
policy independently. That means policies can be set at an enterprise level. As compos-
ite applications grow in number, registries and service governance will become critical
aspects of the integration infrastructure.
COmpOsITe applICaTIOn plaTfOrms
The development paradigm of composite applications involves assembling existing
components, or services, into a new system. However, it will often be necessary to de-
velop new services and to integrate composite applications with other enterprise sys-
tems. Therefore, both the integration vendors and the application server vendors are
beginning to position their platforms asand evolve their offerings intocomposite
application platforms. Naturally, the integration vendors take an integration-centric
view, while the application server vendors focus on the development aspects. Never-
theless, a common set of composite application platform characteristics is emerging.
These characteristics include custom development capabilities, integration capabili-
ties, process orchestration, and user interface functionality.
Depending on the vendor, the composite application platform may be based on an ap-
plication server that also includes integration and orchestration capabilities, or it may
have an integration platform as its foundation, with process management capabilities
and a registry for managing services. Because it is an emerging technology segment,
the offerings are extremely varied.
Process Integration Services
Process integration services enable organizations to automate and optimize business
processes that cross application and organizational boundaries. This helps organiza-
tions reduce operating costs, gain real-time visibility into end-to-end business process-
es, and increase business agility.
68
Chapter 4
Guide to Integration Technologies
While underlying integration technologies and composite application development
can help make IT more productive and agile, the optimization of business processes
can deliver competitive advantage and significant ROI. As mentioned earlier, in the
1950s, Dr. W. Edward Deming proved that improvements in managing business pro-
cesses lead to improvements in overall business and a significant competitive advan-
tage.
A process-driven approach to integration also improves the alignment between IT and
the organization. It starts with the development of business process models that busi-
nesspeople can understand and review. These models depict a shared understanding
of the end-to-end business process. Using process integration technology, the models
can then be automated. This significantly reduces the chance of misinterpreting the
business requirement or getting the process wrong. Since processes can span depart-
ments, business units, and organizations, no single individual may own or even un-
derstand the end-to-end process. Therefore, process integration tools need to foster
collaboration on a process model and an iterative approach to its design and imple-
mentation.
An organizations business processes may be automated, manual, or collaborative. Fre-
quently, an end-to-end business process includes both automated and manual pro-
cesses. From an integration perspective, each type of process has different require-
ments and therefore calls for different process technologies. The technologies that
deliver process-level integration include business process management for handling
automated (and sometimes manual) processes, workflow management for manual
processes, and groupware or collaboration platforms. Some vendors offer solutions
that combine these capabilities into a single product offering. Some also offer process
simulation and analytics to provide feedback for optimizing processes.
Bpm
The business process management solutions on the market today represent a range
of process integration capabilities to support process automation and optimization.
These capabilities include modeling business processes, automating the processes
69
Chapter 4
Guide to Integration Technologies
directly from the model, and monitoring running processes. Available solutions have
varying levels of support for managing manual processes as part of the overall process
and for using simulation capabilities to optimize processes. Another evolving capabil-
ity is the ability to externalize business rules rather than having them hard-coded into
the process integration software. That way, businesspeople can directly control their
portion of the process and thereby increase business agility. For example, during the
busy season, a manager could raise the dollar amount that would trigger an approval
for an order.
Unlike BPEL, which is gaining mind share as an underlying execution language, no
widely supported standards currently exist for process modeling. One candidate, Busi-
ness Process Modeling Notation (BPMN), is working its way through the standards
process within the Object Management Group (OMG). However, few vendors currently
support BPMN. Most have their own modeling notation with BPEL support for process
execution.
As mentioned, a number of BPM solutions also include simulation capabilities to facili-
tate process analysis and optimization. Process designers can perform what-if analysis
to test and compare variations of the process for better productivity and efficiency.
Some tools also allow performance statistics from running processes to be fed back
into the simulation engine to analyze and optimize processes dynamically. This makes
continuous process improvement possible and is a key to gaining competitive advan-
tage.
BPM solutions can also be used to create composite applications in the form of busi-
ness processes that span multiple systems. The BPM tool can model and implement
the flow of control across services and applications. BPM products on the market today
generally provide additional capabilities beyond simple process orchestration in the
areas of process monitoring, management, and optimization.
WOrkflOW managemenT
Workflow management solutions focus on the efficient coordination of manual tasks.
These solutions have been available for many years and predate most BPM solutions.
70
Chapter 4
Guide to Integration Technologies
However, because end-to-end business processes frequently include automated as well
as manual processes, many integration vendors now offer BPM and workflow manage-
ment in a single platform. Vendors that focus exclusively on workflow management are
rapidly decreasing in number.
BPM solutions vary in the level of support they provide for manual processes. Robust
workflow management involves process modeling as well as the design and develop-
ment of user interfaces to manage task lists, accept or change assignments, balance
workloads, and account for changes in resource availability due to vacations, illness,
etc. Manual workflows often involve approvals, which need to be logged and audited
and require the workflow solution to understand the organizational hierarchy and ap-
proval chain. In addition, different interfaces should be available for managers and the
employees receiving the tasks.
COllaBOraTIOn plaTfOrms
Also known as groupware, collaboration software facilitates interactions among peo-
ple. In contrast with workflow, where the interactions follow established processes with
predefined paths and steps, the interactions in a collaboration scenario are generally
free-form and ad hoc. Although collaboration is not usually considered a mainstream
component of business integration, it is relevant to organizations seeking to imple-
ment integration solutions as part of a strategic initiative to optimize business process-
es. This is especially true as project teams increasingly become physically distributed.
Collaboration is also relevant to B2B, where more specialized solutions enable partners
to work together on product design or development, and other joint initiatives. Col-
laboration platforms include repositories for managing team artifacts, such as docu-
ments and design specifications; version control and security capabilities; and features
for managing email threads and discussion groups.
Business Optimization Services
Business optimization services enable organizations to align business strategies with
coordinated actions. The goal of BPM is to streamline processes, but optimizing indi-
71
Chapter 4
Guide to Integration Technologies
vidual processes does not necessarily lead to optimization at an organizational level.
The goal of business optimization is to optimize at the organizational level and to align
all parts of the organization to enhance business performance.
Business optimization services include technologies that enable business performance
management. As a discipline, business performance management includes methodol-
ogysuch as Six Sigma, Balanced Scorecard, and Leanbest practices, and technol-
ogy infrastructure.
Several developments have emerged on the technology front to support business
performance management. The most significant of these are dashboards for execu-
tives, line-of-business managers, and IT operations staff that provide real-time visibility
into the organizations operations and associated KPIs. This capability is provided by an
emerging technology called business activity monitoring.
The well-known analyst firm Gartner coined the term business activity monitoring in
2002 and defines BAM as the concept of providing real-time access to critical busi-
ness performance indicators, along with the supporting information to improve the
speed and effectiveness of business operations. Advances in integration technology
have allowed organizations to instrument applications to deliver real-time informa-
tion associated with significant business events, such as the receipt of an order. BAM is
the technology that monitors the events, correlates them to identify trendssuch as
a sudden, unexpected drop in order volumesprovides real-time analytical capabili-
ties and management dashboards, and alerts users when exceptions occur. Some BAM
solutions include capabilities to predict when exceptions are imminent by comparing
current operating metrics with historical norms.
BAM solutions are emerging from BPM, integration, database, business intelligence,
and systems management vendors. The market for BAM is a quickly evolving sector,
and one that is not yet fully defined. The key feature requirements for business perfor-
mance optimization include dashboards, statistical analysis of user-defined KPIs, auto-
mated and dynamic performance thresholds, and advanced correlation and predictive
monitoring to help managers identify developing concerns and react before they be-
come full-blown crises.
72
Chapter 4
Guide to Integration Technologies
Ultimately, business performance optimization must encompass all levels of the orga-
nization, both on the business side and within IT. For example, an emerging category
of solutions from software management vendors is business transaction management.
This technology enables systems administrators to view the status of end-to-end trans-
actions as they cross systems on a business process level so they can better understand
the context of the transaction.
Compliance and Industry Solutions
The previous categories all represent integration infrastructure services. They are the
technologies to use when creating business solutions. The Compliance and Industry
Solutions section of the Roadmap encompasses business applications that utilize the
integration infrastructure technologies to provide specific business applications.
The difference between these solutions and traditional business applications lies in
their process-centric nature and in the fact that the processes typically span multiple
underlying systems, organizational groups, and other entities. In other words, these are
inherently integrated applications. Compliance and industry solutions require an inte-
grated infrastructureincluding process monitoring, tracking, and auditingbecause
none of the individual participating systems has the total end-to-end view.
Compliance is an example of a business requirement that involves integration across a
diverse set of areas and has a strong monitoring and auditing requirement. Many inte-
gration vendors and systems integrators now offer Sarbanes-Oxley, Basel II, and Health
Insurance Portability and Accountability Act (HIPAA) solutions that make it faster and
easier for organizations to implement solutions to comply with government regula-
tions. Similarly, integration vendors and consultants are offering solutions for specific
industriesincluding healthcare, manufacturing, finance, and retailas well as solu-
tions for supply chain management and customer relationship management.
Organizations seeking to implement new business solutions that require integration
will focus on vendors in this category.
73
Chapter 4
Guide to Integration Technologies
Management and Security
Every level of the integration infrastructure requires management and security. Emerg-
ing management offerings include third-party solutions for application integration
platform management, Web services management, and end-to-end transaction man-
agement at a business process level.
With the growing adoption of SOA, the need for Web services management has be-
come increasingly important. Web services management frameworks help users deal
with issues such as registering Web services across the enterprise, monitoring those
Web services in operation, and determining problems, as well as related areas such as
governance and security. Without management, the ad hoc creation of Web services
across the enterprise can quickly result in chaos, with no one, for example, having a
complete understanding of what is available, who is using what service, and which
services are secure.
Security is another extremely important aspect of integration, and integrated security
solutions are beginning to emerge. Solutions that include digital certificates and en-
cryption are especially important for B2B security. Where standards are lacking, vendor
solutions are filling in the holes, especially in the area of Web services.
Architecture and Development
The ebizQ Integration Roadmap helps explain the types of integration technologies
available. Some vendors offer only one type of technology, while others offer platforms
that incorporate many aspects of the Roadmap in their solutions.
Whether an organization is partnering with one vendor or several, taking a strategic
approach to architecture and development is a key element for success in implement-
ing the technologies represented in the Roadmap. Organizations that take a tactical
approach to developing integration solutions risk suffering the same problems expe-
rienced with developing distributed applications: multiple redundant technologies
used for different implementations, resulting in increased maintenance costs and a
more fragile architecture. Organizations that take a strategic approach to developing
74
Chapter 4
Guide to Integration Technologies
the architecture start with an overall plan and then implement it on a project-by-proj-
ect basis. This prevents the purchase of redundant technologies for different projects,
avoids overlaps in integration technologies, reduces training and maintenance costs,
leverages IT investments, and increases business agility. Rather than requiring individ-
ual project teams to research and choose their own integration technologies, they can
simply use what is already in place and focus instead on adding business functionality
and value.
A well-defined and integrated technical architecture also increases flexibility in the de-
velopment phase. If the business services conform to standard interfaces, they can be
written in a variety of languages, enabling organizations to leverage skill sets within
the organization. For example, services written in Java can be part of the same compos-
ite application as services written in .NET.
Organizations that wish to take a strategic approach to developing an integrated in-
frastructure should establish a centralized group. Some organizations may place this
group within their Enterprise Architecture Group, if they have one. However, since inte-
gration requires specialized skills, many organizations are establishing an Integration
Competency Center (ICC). Chapter 6 focuses on best practices for creating an ICC.
Conclusion
Integration is a journey consisting of many tactical implementations. Organizations
that take a strategic approach to building an integration architecture will be better
able to avoid integration mistakes, control redundancy, lower the cost of ownership,
increase ROI, and ultimately achieve greater business agility. A key element of this is
understanding the role of different integration technologies and their context within
the overall integration infrastructure. The ebizQ Integration Roadmap can guide you in
this process.
75
Executive Overview
S
ervice Oriented Architecture (SOA) is transforming IT, so it is important to under-
stand the relationship between integration and SOA. This is not a straightforward
task, however, because definitions and interpretations of SOA abound.
From a software engineering perspective, SOA is simply an approach to software de-
sign that involves assembling systems from reusable components (services) that may
originate from different sources and underlying technology environments. At this lev-
el, SOA is often associated with Web services. However, Web services address only the
low-level interactions between services; they do not address the grander architectural
and design philosophies of SOA. In other words, SOA is about much more than just
Web services.
From a broader IT perspective, SOA suggests a fundamental shift in how an organiza-
tion implements business systemswith attendant changes in technology, method-
ology, and organizational structure. An even broader interpretation is that SOA marks
the end of monolithic enterprise applications and the beginning of more flexible and
adaptable business process-centric applications.
While definitions of SOA vary, there is unanimous agreement on SOAs value and prom-
ise. The past decade has seen unprecedented leaps in computing. More and more as-
pects of a business use computers. Additionally, the Internet has extended system inter-
faces to include connections with trading partners. Not surprisingly, this has resulted in
an unruly explosion of stovepipe applications, databases, and electronic connections.
The challenge for IT today is to create and support an infrastructure that can harness
these diverse assets and deliver on new business requirements more effectively. SOA
provides the basis for this infrastructure and offers the promise of tangible business
benefits. SOA also makes it possible for organizations to leverage IT assets so they can
rapidly deploy new business solutions as needed.
Integration and SOA
5
76
Chapter 5
Integration and SOA
With its promise of transforming the way systems are developed, SOA will benefit how
organizations approach integration. At a basic level, the growing adoption of Web ser-
vices will improve system-to-system interoperabilitymaking it simpler to connect
systemswhile the componentization of business services will make it easier to cre-
ate new business solutions from existing applications and retrieve information as the
business demands. At the same time, integration technologies play an important role
in the SOA portfolio, addressing needs such as service orchestration, service- and pro-
cess-level monitoring, service management, and the ability to make legacy systems
part of the SOA environment. Clearly, integration and SOA are highly complementary,
with SOA making it easier to integrate systems and business processes, and integration
technologies providing the essential technical infrastructure services required for SOA
implementations.
Unlike previous IT revolutions that were accompanied by massive change, SOA enables
an evolutionary approach and the opportunity for organizations to move forward in-
crementally. The key to success is to plan strategically at an enterprise level and then
implement tactically, with the appropriate best practices and governance policies and
procedures in place. This strategy delivers benefits along the way, making it easier to
justify SOA investments.
This chapter explores the benefits, design concepts, technologies, and best practices of
SOA, and shows how they relate to integration.
77
Chapter 5
Integration and SOA
Case Study 5-1
Company: WRA Electronics
Company Description
WRA Electronics is a pseudonym for a Fortune 500 global provider of electronic components
and computer products. The company serves as a supply channel partner for more than 500
suppliers and 150,000 original equipment manufacturers, contract manufacturers, and com-
mercial customers. Headquartered in New York, WRA employs more than 10,000 people in
more than 200 locations around the world.
Business Challenges
As a player in the global electronic equipment market, WRA is required to comply with RoHS,
a European industry directive restricting the use of six hazardous substances in electrical and
electronic equipment. As part of the directive, WRA needed to make all relevant information
about its products accessible to its customers for quoting, ordering, and purchasing decisions.
However, different business processes (e.g., quoting and ordering) employed different sub-
systems running on different platforms and diverse technologies, and the global components
data was located in an Oracle database.
How Integration Addressed the Challenges
WRA adopted an SOA strategy to address its requirements. Using webMethods as its SOA
platform, WRA was able to link its disparate systems using Web services and to provide a
service-oriented interface for consumer applications. The webMethods solution also provides
connectivity to WRAs mainframe-based systems that currently cannot interoperate with Web
services. The RoHS Web service provides data from the Oracle database and plugs into the ex-
isting business processes for quoting and ordering. Any differences between the platforms are
bridged using the Web service interface, and the webMethods platform performs all necessary
mappings and transformations.
Bottom-Line Business Benefits
WRA is now able to comply with the RoHS mandate. The new system provides global en-
terprise visibility into the companys products and parts along with RoHS information. Each
month, more than 500,000 transactions pass through the companys internal Web services
integration environment, and the RoHS Web service is utilized more than 80,000 times.
In addition to complying with the directive, WRA has achieved significant cost savings and
efficiency improvements with its SOA approach. The system has enabled the automation of
numerous process steps, saving time and errors. Company sales personnel have also been able
to help customers make more informed purchasing decisions, increasing customer satisfaction.
By providing its customers with quick and easy access to RoHS-related information, WRA has
gained a competitive edge in the market.
78
Chapter 5
Integration and SOA
Why SOA?
SOA provides many benefits to both IT and the orga-nization at large. The most im-
portant benefitand the one most often cited in relation to SOAis reusability. SOA
allows organizations to do more with the resources at their disposal. It promotes reus-
ability by enabling services to be shared by different applications running on different
platforms. Reusability creates a competitive advantage by providing greater flexibility
in the way computer systems can be used to support the business.
Figure 5-1. Flexibility of SOA through reuse
For example, many organizations have stovepipe applications that address specific ar-
eas of business functionality, such as order processing, invoice handling, or customer
relationship management (CRM). Each application has its own customer data and its
own view of the customer. And each has local mechanisms for retrieving information,
such as customer details, that were developed in the environment and technology
specific to the application platform. In an SOA, these duplicated routines can be re-
79
Chapter 5
Integration and SOA
placed by a common reusable service (for example, FindCustomer) that delivers the
same functionality. Because there is just one copy of the FindCustomer service, chang-
es need only be made in one place. This reduces application maintenance costs and
enables new changes to be accommodated more quickly and easily.
Another benefit of SOA is increased agility when addressing business opportunities or
changes in business requirements. By reusing common steps and developing only the
new steps that are required, organizations can more easily modify or enhance business
processes and applications when the need arises. For example, a company that offers
home loans may have a loan approval process consisting of numerous SOA services.
If the company now wants to offer car loans, it can reuse many components from the
existing implementation, such as checking current account details or market interest
rates. This increased agility translates directly into increased competitiveness. Thus, or-
ganizations that master SOA will gain an advantage over those that do not.
A third benefit of SOA is the reduced complexity from a more orderly enterprise archi-
tecture and development approach. Modular services enable a less-complex, divide-
and-conquer approach to software development. This enables SOA to accelerate the
deployment of new application functionality, decreases the testing effort by making it
easier to test individual components independently, and lowers implementation risks
through the reuse of functionality that is already quality-tested and proven.
Another benefit of SOA is increased IT adaptability. Packaged application implementa-
tions, custom-built solutions, and system changes resulting from mergers and acquisi-
tions can be integrated more quickly and easily. SOAs inherent platform-independent
approach provides organizations with more flexibility to use the software and hard-
ware of their choice. This allows them to engage in a multisource strategy and reduces
the threat of vendor lock-in. For solutions developed in-house, it enables programmers
to use their existing technical skill sets.
80
Chapter 5
Integration and SOA
Figure 5-2. Adding new functionality easily
Best of all, these benefits can be realized quickly. SOA is ideally suited for incremental
deployment, with investments made step-by-step for individual projects. Returns can
also be realized incrementally. This eliminates the need for costly and risky big-bang
software implementations, reduces investment risk, and delivers value more quickly.
Thus, SOA becomes a far more palatable investment for organizational decision mak-
ers.
SOA Explained
Although the term Service Oriented Architecture first emerged in the 1990s, the spirit
of SOA existed long before that. The underlying principles of SOAcomponentiza-
tion, encapsulation, separation of interface from implementation, loose coupling, and
so onhave been promoted since computings infancy. These design principles were
used in the days of centralized computing, when the focus was on IT resource optimi-
zation.
81
Chapter 5
Integration and SOA
Then the landscape changed. On the technology side, the arrival of distributed com-
puting offered great productivity and price performance, and the emergence of the In-
ternet promised ubiquitous connectivity. On the business side, IT was increasingly seen
as something that could be used for business advantage rather than just back-office
operations. But these changes introduced considerable challenges in linking together
different technologies and platforms to utilize the increasingly diverse IT infrastructure
to meet new business needs. The advent of object- oriented computing paved the
way for SOA by introducing the concept of having discrete program components with
standard inputs and outputs that could be used to assemble applications.
So what exactly is Service Oriented Architecture? From a software engineering stand-
point, SOA is an IT architecture based on the delivery of reusable, well-defined busi-
ness services underpinned by IT components in such a way that the providers and the
consumers of the business services are loosely coupled and have no knowledge of one
anothers technologies, platforms, or locations.
An explanation of the term business services is helpful here. With SOA, IT compo-
nents are formed into services that are relevant to a particular business operation. For
example, the service for FindCustomer might involve code that locates the customer
number by searching for the customers name in one system, then more code that
uses the customer number to query the customers service subscriptions in another
application, and yet more code to retrieve the organizations interaction history with
the customer from a third database.
A service consists of a well-defined service interface and the service implementation.
The service interface describes how to call the service. It specifies, among other things,
the location of the service and the information required to make a request to the ser-
vice and get a response. This is where Web services provide an ideal fit for SOA. The use
of Web services ensures a standard protocol for communication between the service
consumer and provider and a standard way to define a services interface. It is basically
an electronic description of how to call the service from other programs.
The service implementation is the actual code that fulfills the functionality of the ser-
vice. It is the logic that resides on computers somewhere on the network and executes
82
Chapter 5
Integration and SOA
when the service is called, subject to appropriate security constraints. The service im-
plementation is inherently platform-dependent. However, SOA is not concerned with
how services are implemented. Architecturally, it does not matter, for example, whether
the FindCustomer service is written in Java, C#, or COBOL. The only thing that matters
is that the service fulfills the behavior specified by its interface.
Another important term used when defining SOA is loose coupling. Services that are
loosely coupled can be invoked by any application on any platform in any location. All
the calling application needs to know is how to invoke the service, and any inputs and
outputs required. This enables the interoperation of different applications and services
written in different languages with different development tools and platforms. Loose
coupling also means that one service can be replaced by anotherprovided it has the
same signature (inputs and outputs)without impacting any of the other connected
services or requiring them to be modified. That means incremental changes to a sys-
tem can be made without having to redevelop the entire application. The dynamic
nature of SOA is another important distinction from more traditional application de-
velopment approaches.
A final consideration when thinking about SOA is granularity. Granularity refers to
the level of functionality provided by a service. A fine-grained service (such as Veri-
fyZipCode) performs a very specific and discrete function, while a course-grained ser-
vice (such as ProcessInvoice) incorporates more functionality and typically maps more
closely to the notion of a business service. This distinction sets SOA apart from previous
computing paradigms by providing a common language for IT and the line of busi-
ness to discuss computing requirements. Rather than talking about low-level bits of
programming code, SOA allows application requirements to be expressed in terms of
modular, reusable business services. At a technical level, SOA enables services to be
made up of other services, which makes it possible to assemble course-grained servic-
es that map to useful business functions. An important goal when designing services
is to encapsulate business functionality at an appropriate level of granularity so it can
be reused across different solutions.
83
Chapter 5
Integration and SOA
The Role of Web Services
As mentioned, SOA is often discussed in conjunction with Web services, but the two
are not synonymous. In theory, SOA does not require Web services, and simply imple-
menting Web services does not result in an SOA. However, Web services are the first
standard for service-oriented computing to gain the near-universal support of soft-
ware vendors, end-user organizations, and standards organizations. By standardizing
essential elements of SOA, Web services remove the risk of having to bet on a particular
technology (as was the case with CORBA in the days of distributed computing). As a
result, Web services and their related standards are driving the widespread adoption
of SOA.
The Web services standards define XML-based protocols and interface descriptions
that enable services to interact. Basic Web services standards include Simple Object
Access Protocol (SOAP), Web Services Description Language (WSDL), and Universal De-
scription, Discovery, and Integration (UDDI).
XML, or Extensible Markup Language, is a World Wide Web Consortium (W3C)-recom-
mended standard for defining different types of data. XML provides a text-based way
to describe and apply a tree-based structure to information. It has several advantages
over previous approaches to describing data: It is readable by humans and machines, it
is platform-independent, and it has the flexibility to handle arbitrary types of data and
data structures. Propelled by the adoption of the Internet and open standards, XML has
emerged as the critical standard for data definition and document markup.
SDAP is a lightweight protocol intended for exchanging XML-based messages in a
decentralized, distributed environment, normally using HyperText Transfer Protocol
(HTTP), the protocol used between Web servers and browsers on the Web. HTTP was
chosen as the primary application-layer protocol for SOAP because it works well with
todays Internet infrastructure and is firewall friendlyunlike other distributed com-
puting protocols, which are generally filtered out by firewalls.
84
Chapter 5
Integration and SOA
A SOAP message consists of a one-way transmission from a SOAP sender to a SOAP
receiver. Applications can use SOAP to create more complex interaction patterns, in-
cluding request/response and multiple, back-and-forth (conversational) exchanges.
The most common pattern is the remote procedure call (RPC) or request/reply pat-
tern, where one program (the client) sends a request message to another program (the
server), and the server immediately sends a response message to the client.
WSDL is an XML-based format for describing Web services. WSDL defines the public
interface to a Web service, which describes how to communicate with the Web service
as well as the operations and messages supported by the Web service. WSDL is typi-
cally used in combination with SOAP. A client program reads a Web services WSDL to
determine what functions are available on the server, and then uses SOAP to actually
call one of the functions listed in the WSDL.
UDDI is an XML-based standard for a Web services registry (i.e., directory) that allows
service providers to publish information about their services and enables potential
service consumers to browse listings of available services (using SOAP messages) to
identify the ones most suited to their application requirements. As such, UDDI is often
described as a Yellow Pages for Web services. Originally intended as a public resource,
UDDI is now more commonly used within organizations as a private registry of their
internal Web services.
Additional Web services standards are being developed in areas such as security, reli-
able message delivery, and transactions. Standards are also being created to define
the business process flow across services (for example, BPEL, which was discussed in
Chapter 4). Unfortunately, as history has taught us, standards alone do not guarantee
interoperability. Vendors may implement subsets of the standards for expediency, or
supersets of the standards for enhanced functionality, but these subsets and supersets
may actually limit interoperability.
85
Chapter 5
Integration and SOA
Figure 5-3. How Web services standards interact
To guard against interoperability issues and the potential slowdown in Web services
adoption that might result, the vendor community has formed the independent Web
Services Interoperability Organization (WS-I). WS-Is charter is to promote the interop-
erability of Web services implementations from different vendors. WS-I creates profiles,
which are implementation guidelines for interoperability, and testing tools to verify
conformance with a profile and WS-I guidelines. Currently available profiles include the
Basic Profile, Attachments Profile, and Simple SOAP Binding Profile. In addition, work is
under way on a Basic Security Profile. WS-I profile conformance is an important con-
sideration for organizations that want to ensure interoperability of their Web services
across different vendor platforms.
86
Chapter 5
Integration and SOA
Case Study 5-2
Company: U.S. Power & Light
Company Description
U.S. Power & Light is a pseudonym for one of the countrys largest power providers. The
company manages a distribution grid that spans 11 states and provides electricity to 5 million
customers.
Business Challenges
Several years ago, the Federal Energy Regulatory Commission mandated that U.S. Power &
Light become a member of a regional transmission organization (RTO). RTOs are indepen-
dent, government-regulated organizations focused on energy transmission and distribution
for regions of the country. U.S. Power & Light needed to meet the RTOs stringent technical
requirements in 10 months, or it risked federal fines of $1 million a day. These requirements
included providing real-time and batch information on capacity, number of units online, power
produced per hour, emissions, and outages directly from the companys power plants. At the
same time, U.S. Power & Light needed to improve its operational efficiency to meet Sarbanes-
Oxley requirements. Redundancies in the existing system required employees to enter the
same information into 15 different systems for some transactions.
How Integration Addressed the Challenges
More than 100 new interfaces were required for the types of data and forms that U.S. Power
& Light would be exchanging with the RTO. The company chose webMethods as the anchor
of an SOA strategy that enables it to publish data in one place to support the RTO require-
ment. With the webMethods platform, all subscribing systems can now instantly retrieve the
information they require. The solution is based on XML schemas, which are translated from
the source system into U.S. Power & Lights internal canonical forms. This enables the data to
be integrated into multiple systems without point-to-point data transformations. With the new
system, U.S. Power & Light sends its RTO hourly schedules of where it will be delivering power
and where it will be pulling power off.
Bottom-Line Business Benefits
The SOA-based webMethods platform has enabled U.S. Power & Light to be development
platform-agnostic for application interfaces. Using Web services, the company can easily con-
nect disparate systems and reuse code when automating new processes. U.S. Power & Light
met its 10-month schedule and avoided $1 million per day in potential fines. In addition, it can
now meet the Sarbanes-Oxley requirements. With all the integration points in its systems con-
nected through a common platform, U.S. Power & Light can easily monitor and document the
system and processes, errors, and other key areas as required for compliance. SOA and Web
services were two key ingredients for this success.
87
Chapter 5
Integration and SOA
Integrations Role in SOA
Although SOA is commonly associated with the creation of new systems, SOA and in-
tegration are also closely linked. A key driver of SOA is interoperability, which of course
is at the core of integration. In driving a greater degree of system-to-system interoper-
ability through the adoption of Web services standards, SOA will ease the challenge
of integration. For reasons explained below, however, it will not replace the need for
higher-level integration capabilities. Still, organizations can increase flexibility and re-
use by taking an SOA approach to implementing integration projects. In essence, SOA
makes integration easier, and integration technologies enable SOA to achieve greater
productivity and utility.
So how does integration serve an essential role in SOA?
For one thing, as stated earlier, Web services on their own do not provide the full spec-
trum of functionality required in an enterprise SOA. Beyond the interoperability en-
abled by Web services, additional capabilities are required to assemble, orchestrate,
and coordinate Web services into more solutions in real-world business scenarios.
For example, it will be a very long time, if ever, before all enterprise data is accessible
in XML format, the data format for Web services. Different applicationsbuilt with dif-
ferent business requirements for different purposesdefine business entities (such as
customers, products, or financial securities) their own way. Consequently, translation
and transformation of data into the native formats of each application is an essential
component of integration today, and will remain an essential component of SOA. Dif-
ferent data representationsin terms of syntax (the format of the data) and semantics
(the meaning of the data)will exist as long as more than one system is involved. Since
so much data is represented in non-XML formats, XML-based translation and transfor-
mation technologies such as XSLT do not suffice. Therefore, integration technologies
with their robust support for complex legacy data formatsare needed to provide
SOA with a wider range of translation and transformation capabilities.
Additionally, forward-migration mechanisms are needed to bring the legacy environ-
mentthat is, the non-XML, non-Web-services-based systems that will continue to
88
Chapter 5
Integration and SOA
constitute the majority of enterprise systems for a long timeinto the SOA fold, since
starting with a clean slate is not a viable option. This forward migration is often ex-
pressed in terms of putting Web services wrappers around legacy interfaces to make
them available to the SOA world.
Though jumping from an existing legacy interface to one based on Web services may
be technically feasible, it may not be desirable from a financial or risk-management
perspective. Creating Web services interfaces for existing systems requires consider-
able development and investment. If the service will be reused often, the investment
will pay off over time. For some implementations, however, getting a new solution up
and running quickly is the prime requirement. When connecting to enterprise resource
planning, packaged, and legacy applications, it may be faster and easier to use existing
application adapters provided by an integration vendor than to recreate access via
Web services interfaces. Using application adapters for tactical solutions while taking
an evolutionary approach to creating Web services from existing applications will al-
low an organization to realize an early return on investment (ROI) from SOA imple-
mentations. Some integration vendors enable a more seamless transition by making
operations exposed through an adapter automatically available as Web services.
Finally, Web services do not provide the requisite infrastructure services for organi-
zations focused on using SOA to support a strategic initiative to optimize business
processes. Process optimization requires managing the end-to-end business process,
which crosses application and organizational boundaries. This is what business pro-
cess management (BPM) systems do. Web services orchestration defines the flow of
control for a composite application, but BPM goes further with real-time monitoring
management, and process optimization through process automation, human work-
flow management, and process simulation.
The relationship between SOA and integration can be summed up as follows:
SOA and integration are highly complementary. On the one hand, SOA simpli-
fies integration by driving interoperability between systems via Web services,
and by componentizing application functionality into business services, mak-
ing systems more accessible to integration. On the other hand, integration
technologies and solutions provide a migration path to SOA from the existing
installed base of pre-SOA systems.
4
89
Chapter 5
Integration and SOA
Interoperability and integration are not equivalent. Web services promote
cross-platform interoperability, but true business integration demands higher-
level capabilities such as process orchestration, process monitoring, data rec-
onciliation, and exception management. These capabilities are necessary ele-
ments of SOA-based business solutions, but they are not features provided by
Web services. Integration technologies and solutions exist to fill these needs.
The disciplines behind integration and SOA-based development are similar,
and best practices developed for integration can be applied to SOA. Organiza-
tions that take a strategic approach to integration understand that success lies
in appropriate methodologies, governance on the use of organizational IT as-
sets, and dedicated organizational structures and roles. These same principles
apply in practice to SOA. For example, the expertise and processes developed
to manage integration interfacesto ensure that they are well specified, meet
required service levels, and so oncan be applied directly to a registry of busi-
ness services. In short, the expertise and competencies developed for integra-
tion give organizations a head start with SOA.
Enterprise-Class SOA
As stated earlier, simply making Web services available on the network does not create
an SOA. For an SOA to be effective in the real world, numerous infrastructure capabili-
ties are neededsuch as data translation and transformation, process orchestration,
and the other integration capabilities described in the previous chapteras well as
capabilities for security, versioning, auditing, management, monitoring, and ensuring
predictable quality of service. The latter capabilities are especially important for en-
abling SOA to address mission-critical business requirements.
It is useful therefore to think of an enterprise-class SOA as one that not only incorpo-
rates the business services underlying the applications delivered via the SOA, but also
provides important infrastructure capabilities for delivering the reliability, scalability,
and manageability that global organizations expect. A coherent SOA infrastructure will
also help manage the complexity of the environment, without which sustainable reus-
ability would not be possible.
4
4
90
Chapter 5
Integration and SOA
Case Study 5-3
Company: State Bank
Company Description
State Bank is a pseudonym for a European bank based in Germany. With a focus on midsize
enterprises, the bank offers its domestic and international customers the financing services
of an internationally oriented credit institution and the investments and services of an invest-
ment bank. State Bank has more than 1,700 employees and over 70 billion in revenues.
Business Challenges
State Bank was introducing SAPs Bank Analyzer, which includes a financial database (FDB).
The FDB was to be the banks new core database, and all enterprise applications needed to be
connected to it. This requirement made State Banks previous interface model obsolete be-
cause it was time-consuming and led to a single point of failure.
How Integration Addressed the Challenges
State Bank decided that adopting an SOA would shorten the development time of the in-
terfaces, making them reusable and easier to maintain while eliminating the single point of
failure. The bank implemented webMethods Fabric as the SOA-based integration layer for
connecting its legacy systems and databases to the FDB, and used XML to map all the data
entering and leaving the integration layer. As a result, all of the banks systems can now com-
municate with the FDB for the SAP solution.
Bottom-Line Business Benefits
The SOA-based integration solution has replaced State Banks obsolete interface architecture
with a flexible, reusable system. After only a three-month implementation phase, the new
platform is already processing 170,000 transactions each month. As a result of its approach,
State Bank can more easily synchronize customer information with the customer master data,
obtain foreign exchange prices in real-time, and update new securities holdings after every
transaction.
91
Chapter 5
Integration and SOA
sOa InfrasTruCTure
Deploying an SOA infrastructure as part of the overall SOA implementation strategy is
vital if organizations are to realize the benefits promised by SOA. Although a tactical
approach might be possible in the early stages of an SOA implementation, long-term
ROI requires a proactive approach to creating the SOA infrastructure.
An enterprise-class SOA infrastructure:
Promotes architectural consistency by ensuring that Web services are de-
ployed in a manner consistent with a service-oriented model
Facilitates the assembly of large-scale, dynamic systems from Web services
Provides a central point for managing and monitoring deployed services
Delivers the performance and robustness required for business-critical sys-
tems through features such as load balancing and automatic failover between
services
Provides mechanisms whereby services can easily locate one another, regard-
less of where they are deployed in the network, and promotes loose coupling
of business-oriented services through location transparency and dynamic
binding
Provides metadata management and registry services to support the reusabil-
ity of services
Provides a framework for creating and deploying infrastructure servicesser-
vices that function between business-level services, such as transformation
and filtering
Allows services to be provisioned dynamically as business needs fluctuate
Provides security across services, addressing needs such as client authoriza-
tion
Offers various levels of loggingfor example, Web services requests and re-
sponses, session-related information, and security eventsto support trend
analysis, exception handling, and problem diagnosis
4
4
4
4
4
4
4
4
4
4
4
92
Chapter 5
Integration and SOA
An important requirement of an enterprise-class SOA infrastructure is that it exist in-
dependent of the business services and applications it supports. As the figure on the
next page illustrates, different Web services containers may have different capabilities
(including no native Web services capabilities at all). In addition, even when different
containers have similar capabilities (for example, support for a specific WS-* standard),
the actual implementations may vary or be incompatible because of version differ-
ences.
As a result, the only way to achieve the appropriate consistency across different service
containers for functions such as security, transactions, service monitoring, and quality
of service is to externalize the SOA infrastructure from the service providers. Conceptu-
ally, this means that instead of consumers calling Web services providers directly, calls
are made through an interception layer that provides a consistent set of infrastructure
capabilities across service providers. In addition to enabling consistency, this layer en-
ables the SOA infrastructure to do things like load-balance across different services
and provide Web services capabilities to systems that are not natively built on Web
services.
Figure 5-4. Enterprise SOA infrastructure
93
Chapter 5
Integration and SOA
As explained, the SOA infrastructure should be non-intrusive to the business services
themselves. It should also be capable of operating natively within supported platforms,
such as an application server, and as an intermediary for nonsupported platforms, such
as a packaged application or a service hosted by a third party). With this flexibility, an
SOA infrastructure can bring all the services in an organizations environment under a
single management scope.
sOa gOvernanCe
Providing the capabilities of an enterprise-class SOA infrastructure requires a combi-
nation of standards, technologies, policies, and procedures. Although the appropriate
technologies are important, corresponding policies and procedures are just as crucial
to realizing the benefits of SOA. As Web services begin to proliferate in organizations
and services are reused in different applications and different areas of the organiza-
tion, governance becomes an essential part of the SOA strategy. So does the need to
enforce policies and procedures throughout the development and maintenance life
cycle of a Web service.
One important area of SOA governance is managing reuse. As experience with object-
oriented programming has shown, if ROI is to be realized, the discipline of reuse must
be designed into the process. Reuse does not happen by accident. When the prolif-
eration of Web services is unmanaged, it leads to duplication, overlap, and an over-
all loss of control that erodes the reusability benefits of SOA. Managing reuse means
architecting services so they can be reused, monitoring how services are reused, and
providing appropriate incentives for reuse. Although these issues are primarily ones of
methodology and process, some vendors provide capabilities to help manage reuse.
For example, by profiling the performance characteristics of a given Web service (such
as average response time), a developer can make an informed decision about whether
to use the service for a specific business requirement.
An additional governance issue relates to policy definition, implementation, and en-
forcement. Consider the FindCustomer service. A company may need to implement a
new business policy that requires customer credit card information to be transmitted
94
Chapter 5
Integration and SOA
in an encrypted format to protect customer confidentiality. The developer of the ser-
vice can easily ensure that the service receives and delivers only encrypted informa-
tion. The challenge is communicating and enforcing the policy change with individuals
who are using the earlier, unencrypted version of that service.
A centralized registry helps support the definition, implementation, and enforcement
of service policies. Putting policy rules in a centralized governance repository makes
them much easier to enforce across applications and platforms. For example, develop-
ment policies can be enforced when a service is registered, and run-time policies can
be enforced when a service is invoked. Because of this, the UDDI registry is increasingly
viewed as more than just a place to catalog services; it is also a strategic asset in the
SOA infrastructure from a governance perspective.
Enforcing security policies across services is another important governance issue. A
Web service may be created for use within a single department but then adopted by
other users, including users outside the organization. Security must be maintained
across users. An ad hoc approach to Web services management provides no mecha-
nism for implementing a security policy and enforcing compliance across the organi-
zation.
A final issue relates to the operational monitoring and management of Web services
from a service-level perspective. A Web service may be used by applications across
the enterprise that have different quality-of-service expectations regarding availability,
performance, and security. It may be possible to monitor the performance of the ser-
vice in isolation, but the more important requirement is whether the service satisfies
the service-level expectations of a particular user.
When issues such as these are not addressed at an enterprise level, and tactical Web-
services-based solutions begin to proliferate without effective governance, the ben-
efits of SOA rapidly decline. As such, the more progressive software vendors offer not
only technology solutions but also a broader portfolio of capabilities that includes best
practices and methodologies for helping organizations realize their SOA strategies ef-
fectively.
95
Chapter 5
Integration and SOA
Best Practices in SOA
By now it should be clear that SOA is an enterprise strategy, not a technology or a tacti-
cal solution. To reap the rewards of SOA, organizations must implement an enterprise-
level approach. Best practices include the following:
Create an enterprise SDA group. This group is responsible for defining the enterprise
SOA architecture and roadmap, and managing the SOAs incremental implementation
by working with project groups across the organization. The enterprise SOA group is
similar in function and purpose to the Integration Competency Center (ICC), as de-
scribed in Chapter 6, and may evolve as an outgrowth of the ICC or the Enterprise Ar-
chitecture Group, if one exists.
Manage reuse. As mentioned earlier, reuse must be managed if any benefits are to
be realized. Among other things, managing reuse requires planning and designing for
reuse, maintaining appropriate documentation and metadata about reusable assets,
having a registry of the services available for reuse, and providing active administra-
tion. Policies and incentives for reuse are also important.
ControI redundancy. Duplication of technical and business services drastically in-
creases maintenance costs and complexity. Redundancy usually results from poor
governance and a temptation to take shortcuts. These can be avoided with strong en-
terprise controls and best practices.
DeveIop SDA competency within the organization. Loosely coupled Service Ori-
ented Architectures provide maximum business agility. However, it is not enough to
provide programmers with an integrated development environment for creating Web
services. Programmers need to understand how to design and build systems that fol-
low and take advantage of the new paradigm. For example, developers should under-
stand how to design services at the right level of granularity to maximize reuse. For
long-term success, organizations need to invest in developing a core competency in
SOA design and implementation.
96
Chapter 5
Integration and SOA
Conclusion
SOA is the undisputed path for creating an agile IT infrastructure. Although there are
many important technology considerationsincluding the implementation of the in-
tegration technologies represented in the ebizQ Integration Roadmap (Chapter 4)
SOA is about more than technology. The broad, successful adoption of SOA requires a
proactive, managed approach that has implications for an organizations development
practices, its organizational structures, and its approach to investment in applications
and software infrastructure. Taking the proper strategic approach requires an initial
upfront investment, but the benefits accrue as the number of deployments grows.
Over the long run, a strategic approach offers a much higher ROI than what might be
experienced with quick and easy tactical solutions.
97
Executive Overview
A
major theme of this book is that integration and Service Oriented Architecture
(SOA) are not shrink-wrapped solutions, nor are they simply about technology. If
an organization wants to reap all the rewards that integration has to offer, it needs to
follow best practices, avoid common pitfalls, and understand how to structure the or-
ganization for success. Risks include ignoring data quality issues, lacking clear require-
ments or well-defined project plans, not having a cohesive organizational structure,
and lacking processes and procedures for implementing integrated solutions.
Integration skills are specialized, in demand, and in short supply. One of the more im-
portant best practices to emerge is the notion of an Integration Competency Center
(ICC) within the IT organization. An ICC provides an environment in which critical skills
can be honed, knowledge and experience from previous projects can be consolidated
and reused, integration processes can be developed and tested, and governance can
be applied to reusable integration assets and business services.
Most organizations will spend considerable sums on implementation and consulting
services. However, integration needs to become a core competency within the orga-
nization so that new solutions can be easily implemented on an ongoing basis. This
chapter addresses some common integration pitfalls, explains the importance of tak-
ing a strategic approach to integration, examines various models for establishing an
ICC in an organization, defines roles and responsibilities, and identifies the critical re-
quirements for successful integration projects.
Common Pitfalls
By taking a few precautions, organizations can avoid the pitfalls that jeopardize an
integration projects success.
Best Practices for Integration
6
98
Chapter 6
Best Practices for Integration
Data quaIity issues. Integration is pointless if the underlying data quality is poor. As
the adage goes: garbage in, garbage out. Before integrating systems, organizations
should first assess the quality of the original data to determine whether it meets the
usage needs elsewhere. Some level of data cleansing in the source systems is often
necessary to avoid downstream exceptions that reduce the efficiency of an automated
integration process.
Lack of cIear requirements. When an integration project is started while an appli-
cation is still being developed or customized, the integration requirements become
a moving target that is much harder to hit. Before beginning an integration project,
organizations should first ensure that the source systems and interface requirements
have solidified. With integration, the devil is in the details, meaning that even minor
changes to the requirements can entail significant rework of the design and solution.
Poor project pIanning. Organizations frequently fail to plan for the long lead times of-
ten associated with hardware and software procurement or firewall and other security
changes. Consequently, dependency-related delays and scheduling issues are com-
mon. This pitfall can be avoided by working closely with procurement, security, and
operations management to ensure a smooth process. Another aspect of poor planning
is failing to coordinate and establish realistic deadlines when integrating with trading
partners. Organizations should work closely with their trading partners to establish
realistic project timelines, milestones, and testing processes that factor in the compli-
cations of dealing with external entities. They should also communicate frequently to
ensure that all parties are on track with the schedule and milestones. When kicking off
a project, it is a good idea to meet key trading partner contacts in person. This often
reduces the communication barriers during the project.
Poor coordination between I7 and business stakehoIders. Inadequate communi-
cation between IT personnel and key stakeholders in the business opens the door for
future scope changes. This is a potential issue on all projects, but integration projects
are particularly vulnerable because of the number of related parties involved. Organi-
zations should identify the key stakeholders and decision makers for all the interested
parties, and establish a formal process to obtain sign-off.
99
Chapter 6
Best Practices for Integration
Lack of deveIopment standards and guiding principIes. Naming standards, config-
uration management standards, and code promotion standards are critical for agility
and reuse. Organizations need to establish standards and guiding principles early to
ensure a consistent and orderly solution.
Taking a Strategic Approach to Integration
Historically, most integration projects have been undertaken as tactical projects, or as
tasks within a larger implementation project. This is quickly changing because of the
inefficiency and cost of taking a piecemeal approach to integration. Treating each inte-
gration requirement on a project-by-project basis does not afford the opportunity for
reuse from one application to the next. It also increases maintenance costs, since there
is no opportunity for reuse and no way to gain economies of scale.
A tactical approach also inhibits the development of integration skills in an organi-
zation. Although application developers have been doing integration for many years,
most of this has been done in a point-to-point, application-to-application manner us-
ing custom code, application- specific programming interfaces, or flat-file data trans-
fers. As a result, integration skills tend to be specialized and scattered throughout an
organizations IT community rather than centralized in one group. Organizations today
need individuals with the skills required by more sophisticated software and integra-
tion platforms that offer built-in support for integration-specific requirements such as
transactions, auditing, business process modeling, and process automation. However,
few organizations have invested in the resources required to support these new tech-
nologies.
The increasing complexity of integration requirements is also fueling the push toward
a strategic approach. Rather than focusing simply on linking two or more applications
together, integration today seeks to automate process flows both within the organiza-
tion and across organizations (for example, integration with customers and suppliers).
If organizations want to more efficiently build and maintain the complex integration
processes that their businesses demand, they can no longer treat integration as a series
100
Chapter 6
Best Practices for Integration
of heterogeneous, project-driven tasks. Organi-zations that take a strategic approach
to integration by creating a separate organizational entity to focus on integration is-
sues and promote best practices throughout the organization will benefit through
lower maintenance costs, greater skills development, and increased business agility.
Organizing for Success
When it comes to creating an organizational structure to support integration, there is
no one-size-fits-all approach. An integration team can range from a few resources to
several dozen and can vary in the level of services offeredfrom simply developing
guidelines and best practices to serving as a full-scale shared services provider that
offers hosting and administration for all of the organizations integration needs. Re-
gardless of the teams size or the services offered, best practice recommends that an
organization create an Integration Competency Center. The actual composition and
services of the ICC will depend in large part on the IT organizations infrastructure. That
is, the ICC model can be centralized, distributed, or a hybrid of the two.
In the distributed model, an organizations IT resources are divided along functional
areas, product lines, business units, or geographical units. IT groups operate fairly au-
tonomously, and there is little coordination among them. In this model, the ICC estab-
lishes best practices that standardize processes (such as development methodologies
and naming conventions) across project teams.
Although the ICC may not be able to enforce policies and standards in a distributed
IT organization, it can provide a set of integration guidelines for the independent IT
organizations. This increases the likelihood that individual integration projects are ar-
chitected properly. The ICC may also make technology recommendations, though ul-
timately the project teams implement the technology and thus retain a fair degree of
autonomy. The benefit of this model is that knowledge is shared and some procedural
reusability is achieved.
101
Chapter 6
Best Practices for Integration
In a variation of the distributed model, the ICC defines not only the standard proce-
dures but also the standard integration technologies that project teams should use. As
before, however, the actual implementations are done by the project teams. Because
the integration platform is not under the ICCs control, the ICC needs to offer services
such as capacity planning, disaster recovery planning, and deployment assistance to
the external integration teams. The ICC basically acts as an internal consulting group to
the distributed IT organizations. This scenario provides a greater degree of consistency
than the previous one.
In the hybrid model, some of the integration implementation workload shifts to a cen-
tralized group of development resources. As before, the ICC establishes the methodol-
ogy and software standards. However, in this model it also contributes to the develop-
ment of shared integration assets, with resulting benefits to reusability.
In the centralized model, the ICC acts as a full-service provider. Not only does it set
the standards, but it also delivers architecture, design, development, and operations
support services. All or most of an organizations application development and IT infra-
structure is consolidated into one organizational IT group. A full-service ICC typically
performs the following functions:
Running the integration pIatform as a pubIic utiIity. A centrally run and
administered integration backbone has been likened to a public utility. Just as
a telephone can be plugged into a wall jack and receive phone service, devel-
opers can plug systems into the integration backbone and tap into the stream
of data flowing through the integration platform. In a highly centralized IT or-
ganization, the ICC is responsible for building and maintaining the integration
backbone.
Dffering standard interfaces to organizationaI systems. The ICC builds and
maintains a library of standard interfaces into enterprise systems that applica-
tion developers can reuse in their projects.
Providing data center operations for integration projects. The ICC main-
tains the shared integration infrastructure and allocates development, test,
and production environments for integration projects. It also assists the IT
operations staff with production support and monitoring once the integra-
tion project is in production.
4
4
4
102
Chapter 6
Best Practices for Integration
Providing integration deveIopment services. The ICC acts as an internal
consulting group to project teams, providing architecture, design, and devel-
opment services either directly through its own staff or by contracting with
preferred systems integration partners and off-shore development compa-
nies.
Maintaining the enterprise information modeI. In an environment where
multiple integration processes publish and subscribe to data, the ICC must
develop and maintain a business object model (also referred to as a canonical
data model) of all the documents and data to be exchanged through the in-
tegration platform. This role is critical to maintain data consistency and a high
level of reusability.
A centralized model provides several benefits. Policies, procedures, and standards are
easily set and administered. Additionally, having all the services under one roof and
point of control maximizes the opportunity for reuse and consistency. The downside of
this model, as with any centralized service organization, is the increased effort required
to effectively manage and respond to the project teams commitments and priorities.
In the absence of a formal ICC, project teams operate entirely independently, establish-
ing their own development practices and making one-off technology choices. Allow-
ing project teams to innovate without constraints may have some benefits. But this
scenario runs counter to the SOA philosophies of interoperability, governance, and re-
use. As such, it might be applied to a select subset of projects but seldom across-the-
board.
4
4
103
Chapter 6
Best Practices for Integration
Case Study 6-1
Company: MK Healthcare Products
Company Description
MK Healthcare Products is a pseudonym for a Fortune 50 manufacturer of healthcare prod-
ucts. The company was founded over a century ago and currently has more than 200 operat-
ing companies grouped into segments focusing on pharmaceuticals, consumer products, and
medical devices and diagnostics. MK Healthcare Products employs approximately 109,000
people in 57 countries and sells products throughout the world.
Business Challenges
MK Healthcare Products is well known for its highly decentralized management structure. A
combination of organic growth, acquisitions, and a global market has increased the need for
collaboration of business processes, applications, and data across the organizations multiple
operating companies. In the past, the operating companies developed new processes and
services individually, without a common platform, common technology, or business processes.
Each operating company had its own integration strategies, architecture, tools, and standards.
This often resulted in systems that could not integrate easily with one another, and it signifi-
cantly limited the organizations ability to maximize knowledge sharing, reduce redundancies,
and improve business processes.
How Integration Addressed the Challenges
MK Healthcare Products created an Integration Services Group, whose charter was to create
a strategy and platform for integrating business processes and applications. It adopted a total
business integration methodology based on an enterprise application integration solution to
connect the operating companies with one another and with external business partners. The
methodology uses standards, processes, a common design approach, templates, and a philoso-
phy of reuse and covers the entire integration life cycle, from project initiation to deployment.
By moving toward a centralized approach to enterprise integration, MK Healthcare Products
was able to standardize on an integration platform, consolidate hardware, share code and
components, combine education and training initiatives, and leverage best practices across
the operating companies.
Bottom-Line Business Benefits
The centralized Integration Services Group has made tremendous gains for MK Healthcare
Products. To date, more than 120 projects have been implemented using the total business
integration methodology and architecture, resulting in a cost avoidance savings of $9.1 mil-
lion. The new, integrated systems process 300 million transactions per week across all projects,
which equates to $30 billion in transactions annuallyapproximately half of MK Healthcare
Products annual revenues.
104
Chapter 6
Best Practices for Integration
Integration Roles
Regardless of the model chosen, an integration team should incorporate the following
roles and responsibilities:
usiness anaIyst. With integration requirements increasingly being driven by busi-
ness process automation, the integration team or ICC should include resources with
strong business domain expertise and analysis skills. The primary responsibilities of
the business analyst are to understand the business needs, define the related busi-
ness processes and integration requirements, and perform the associated data-level
analysis.
AppIication speciaIist. The application specialist builds the individual components
and Web services from which higher-level integration applications and processes can
be assembled. Typically, the application specialist develops business-level services that
are associated with one or more applications, so he or she needs to have some level
of application domain expertise. Services may be created from scratch (for example,
writing a new customer lookup service), by recombining existing services into new
services, or by wrappering existing functionality with Web services interfaces.
Integration speciaIist. While the application specialist focuses on developing reus-
able business services, the integration specialist is the middleware expert. The inte-
gration specialist develops custom adapters and interfaces, reusable infrastructure
services to facilitate different integration scenarios, and other utility integration ser-
vices to enable the implementation of more complex business process and integration
solutions. The integration specialist also handles the detailed technical engineering
of the implementation, conducts capacity planning and other system-level tests, and
provides some operational support. The integration specialists expertise is generally
centralized within the ICC. In some organizations, the application and integration spe-
cialist roles are filled by the same individuals.
Process-oriented deveIoper. The process-oriented developer bridges the gap be-
tween the business analyst and the application and integration specialists, turning
105
Chapter 6
Best Practices for Integration
business process models into executable business processes that link together the ser-
vices developed by the application and integration specialists. The process-oriented
developer needs both a business-level understanding of the business process and the
technical expertise to assemble and orchestrate the underlying services into a pro-
cess.
Integration architect. The integration architect plays a critical role within the ICC. This
individual defines the overall integration strategy and architecture, performs design
reviews of integration implementations, and mentors the rest of the IT organization
to ensure that the integration software is appropriately leveraged and properly imple-
mented and that integration solutions are architected to take best advantage of the
integration infrastructure. In addition to driving standard integration practices and
methodologies, the integration architect is often responsible for educating the IT com-
munity on the merits of the integration platform and evangelizing the adoption of the
software throughout the organization.
DperationaI support speciaIist. The operational support tasks that the ICC takes on
can vary widely, depending on its scope. Therefore, the number and composition of
the operational support team will vary accordingly. At a minimum, the ICC should in-
clude resources that can provide integration platform deployment guidance to project
teams and data center personnel.
Project manager. The staffing of project managers within the ICC also depends on its
charter. If the ICC is responsible for providing hands-on consulting to the IT commu-
nity, then project managers need to be on board.
ReIationship manager. A strong vendor relationship is critical to the successful adop-
tion of new technologies. The relationship manager acts as the central contact between
the organization and the vendor. He or she keeps the vendor informed of issues that
arise within the organization and notifies the organization about the vendors current
and future product and service offerings.
106
Chapter 6
Best Practices for Integration
Responsibilities of the ICC
The individual responsibilities that an ICC takes on will depend on the organizations
stated goals for the ICC as well as the nature of the organizations IT group. This section
describes the major areas of responsibility for most ICCs and lists possible objectives
for each area.
develOpmenT sTandards and praCTICes
An important function performed by an ICC is developing standards to facilitate reuse
and interoperability. Responsibilities in this area include:
Creating and maintaining integration best practicesfor example, standards
for documenting an integration, naming standards and conventions, proce-
dures for promoting code to production, and instructions for building security
into integration processes
Creating interoperability techniques to standardize the integration of hetero-
geneous platformsfor example, integrating between the webMethods Fab-
ric platform and IBM MQ Series-based applications
Developing architectural patterns for common integration scenariosfor ex-
ample, real-time data lookups, flat-file integration processes, batch integration
processes, and transactional business processes
Developing and maintaining an organizational business object model to al-
low disparate integration processes to interoperate easily
Developing, harvesting, and maintaining reusable integration components
gOvernanCe
Another important function of an ICC, especially in relation to SOA, is in the area of gov-
ernance. Governanceessentially, processes for managing and controlling the use of
enterprise assetsis essential for ensuring reusability, long-term maintainability, and
productivity. Governance responsibilities of the ICC include:
Establishing policies for how an organizations services are built and main-
tained.
4
4
4
4
4
4
107
Chapter 6
Best Practices for Integration
Defining and managing SOA domains, which are sets of services related by
a common business context. The domain owner brokers the relationship be-
tween line-of-business users and the owners of the applications on which the
domain services are built.
Managing the organizations business services repository and the associated
metadata, and ensuring that development practices incorporate the appropri-
ate use of the repository.
Establishing service level agreements for business services.
COnsulTIng and TraInIng
The ICC can act as an internal consulting group to an organizations IT community, pro-
viding expertise on the use of integration tools and techniques. Services include:
Providing architectural guidance and review for individual projects
Developing and delivering custom training materials
Providing in-house first- and second-tier support for integration tools
Assigning individual developers to project teams to build integration processes
CusTOmer advOCaCy and vendOr managemenT
As mentioned earlier, continued success with a chosen technology is often predicated
on a strong relationship with that technologys vendor. To facilitate the relationship,
the ICC should act as the main contact between the organization and the vendor. Re-
sponsibilities include:
Working with the vendor to help market the benefits of both the products and
the ICCs services to the organization
Prioritizing the organizations enhancement and feature requests, and negoti-
ating with vendors
4
4
4
4
4
4
4
4
4
108
Chapter 6
Best Practices for Integration
Participating in vendor early-release programs and planning for the rollout
and migration of new and upgraded projects
TeChnOlOgy leadershIp
The ICC should constantly look for advances in integration technology that can benefit
its organization. The rest of the organization should view the ICC as an expert body for
advice on implementing and deploying integration technologies. Services in support
of technology leadership include:
Writing white papers on emerging technologies and trends
Hosting internal information-sharing sessions
Staying up-to-date on new vendor features and capabilities
Developing prototype applications using new technologies
OperaTIOnal suppOrT
The area of operational support is where the ICCs responsibilities are most likely to
differ from one organization to the next. At one end of the spectrum, the ICC may own
and administer the hardware on which the integration platform is deployed. It would,
in essence, create the physical infrastructure on which the enterprise integration back-
bone is deployed. Project teams would request development and production space on
the shared platform and then turn over their integration processes to the ICC to deploy
and monitor in production. In this scenario, ICC services include:
Procuring and maintaining the physical integration platform infrastructure
Providing production support for integration processes built by project
teams
Allocating development and testing environments for project teams
At the other end of the spectrum, the ICC could simply provide integration platform
deployment guidance to the data center team, which would deploy the integration
processes. ICC services in this scenario might include:
4
4
4
4
4
4
4
4
109
Chapter 6
Best Practices for Integration
Planning capacity for integration platform components
Planning for disaster-recovery scenarios
Developing fail-over and load-balancing environments
Providing production support for integration platform softwarefor exam-
ple, data center personnel would be responsible for hardware and operations
system support, integration developers would support the integration logic
they have developed, and the ICC team would be responsible for supporting
the integration platform software
Creating an ICC
There are two general approaches to forming an Integration Competency Center. An
ICC can be created independent of any one integration project. Or it can be established
in concert with an organizations first integration project and then evolved. We have
found that it is better to take an evolutionary approach and allow the ICC to develop
from the pilot integration project. This increases the opportunities for success, for sev-
eral reasons:
The quality of the standards, practices, and reusable frameworks produced by
the ICC tends to be higher because they reflect the teams actual experience
working with the integration tools in a project setting.
Members of the IT community are more willing to accept the ICCs guidance
because the staff has real-world experience.
Management is more likely to approve funding for the ICC because it is con-
tributing immediately and directly to project deliverables. As a result, the ICCs
return on investment can be achieved quickly.
Once the original project has been completed, the ICC can be transitioned to an inde-
pendent body and can continue refining the methods, tools, reusable architectures,
and infrastructures that were established for the pilot project.
The following example illustrates how to build a full-service ICC using an evolutionary
approach.
4
4
4
4
4
4
4
110
Chapter 6
Best Practices for Integration
pIlOT prOjeCT
In this phase, the ICC team may consist of only one or two resources working with
the pilot project team. The ICCs aim is to help develop standard architectures, devel-
opment and deployment techniques, and the beginning of an enterprise information
model. The ICC staff should include a professional services resource from the integra-
tion platform vendor to facilitate the knowledge transfer of best practices and archi-
tectural techniques to the ICC team.
TransITIOn TO The IndependenT OrganIzaTIOnal unIT
Once the pilot project has been completed, the ICC moves to an independent organi-
zation, gaining resources by transitioning some of the pilot project team members into
the ICC. As this phase begins, it is important to formalize the ICCs charter and clearly
spell out the services that the team plans to offer the IT community. In this phase, the
ICC:
Assesses best practices and standards developed in the pilot project, makes
any changes necessary to render them generic, and publishes them for use on
future integration projects.
Builds architectural frameworks around common patterns of integration, or-
ganizational systems that are widely integrated to, and common technologies
(for example, Web services, MQ Series, and CICS mainframe applications). This
will allow future integration projects to be more productive as well as ease
maintenance and support, since all integration processes will be based on a
common framework.
Begins building the reuse library by harvesting integration components from
the pilot project and making them more generic, if necessary, for reuse on
future projects.
OngOIng suppOrT
As the ICC becomes well established and its standards and practices are deployed, the
team continues to provide support to the organization and:
4
4
4
111
Chapter 6
Best Practices for Integration
Stays current on integration industry trends and practices
Participates in vendor early-release programs to plan for upgrades and to ad-
vise project teams on the use of new features
Continues educating the IT community on integration technologies and best
practices
Continues building the reuse library by gathering components from project
teams as well as assessing the need for common integration processes and
developing them from scratch to offer on future projects
Maintains and grows the enterprise information model to ensure that the in-
tegration backbone provides a coherent canonical model for all the applica-
tions deployed on it
Maintains the business services repository and expands the inventory of ser-
vices available for reuse
Critical Success Factors
Integration can be a complex subject because multiple technologies may be applied
to different business scenarios. Regardless of the type or scope of the integration proj-
ect, the following factors are critical for success:
Link the ICC mission and charter to business objectives. Even when based on sound
technology and executed flawlessly, many a project has been seen as a failure because
it did not deliver on the business objectives that the organization was striving to meet.
Linking mission to business objectives is even more critical when forming an ICC, which
is essentially a cost center to the organization. The ICCs charter and deliverables must
be aligned with the organizations overall objectives. For example, if the orga-nization
is focused on cost savings, the ICC should strive to make integration processes more
efficient, reduce implementation costs, and so on. If the organization is focused on im-
proving customer service, the ICC should concentrate on creating standard integration
processes across all organizational systems that store customer data, building out the
enterprise information model to define a single view of customer data, and so on.
4
4
4
4
4
4
112
Chapter 6
Best Practices for Integration
Identify and communicate integration and ICC successes. For each project, specific
key performance indicatorssuch as reducing cost or implementation time, or im-
proving customer serviceshould be defined. Then the successes should be tracked
and communicated. This will build confidence in the ICC and encourage further orga-
nizational investment in integration infrastructure projects.
uiId a core competency in integration within the organization. ICC resources
should be involved regularly in integration projects so they can continue to increase
their skills and add to the collective knowledge of the organization. Systems integra-
tors and consultants on an integration project must transfer their knowledge to ICC re-
sources. If the ICC does not actively focus on these tasks, integration experience could
become scattered throughout the organization and be difficult to leverage across proj-
ects. Worse, valuable expertise could leave the organization altogether as the systems
integration partners complete their assignments and move on. Its critical to invest in
training to continue building skill sets and knowledge.
Start smaII and buiId on successes. Integration is inherently complex. An organiza-
tion should not begin by trying to boil the ocean. Instead, it should start small and add
integration processes incrementally. Taking on too much too quickly can lead to fail-
ure. It is better to underpromise and overdeliver. In the early stages, the ICC should be
supplemented with experienced professional services resources from the vendor or a
systems integrator to satisfy key project team needs.
Define governance poIicies and procedures. Many benefits of SOA and integration
come from reusing existing organizational assets. However, enabling others to utilize
existing functionality for new solutions requires proactive governance of the reusable
assets. Policies and procedures should be defined, as should service level agreements.
This is a critical success factor for managing reuse.
Dbtain executive support. Integration projects routinely cross organizational bound-
aries. Automating, optimizing, and managing end-to-end business processes or provid-
ing unified access to distributed information can mean stepping on toes. Management
support is critical for helping to break down any barriers to acceptance that may exist
within the organization.
113
Chapter 6
Best Practices for Integration
Conclusion
As organizations invest more strategically in integration and SOA, many decisions and
obstacles are sure to present themselves. To successfully navigate these obstacles, it is
critical to develop and maintain integration competency within the organization. Es-
tablishing and evolving an ICC is a proven approach to promoting best practices; mak-
ing efficient use of resources and skills; and improving the consistency, reusability, and
agility that enable organizations to reap the rewards of integration. Finally, because
of the synergy between SOA and integration and the commonality of best practices,
organizations that have made a strategic investment in an ICC will find themselves
poised for success with SOA.
114
Gary So
VP, Offce of the CTO, webMethods, Inc.
Gary So is Vice President, Offce of the Chief Technology
Offcer, at webMethods, Inc, where he is responsible for
advancing the companys status as a recognized industry
thought leader. Gary has over 10 years experience in
the integration feld, serving previously as a system
architect in corporate IT and as a director of professional
services at Active Software, Inc., before joining webMethods in 2000 and spear-heading
the development of the companys integration methodology. Formerly VP of Strategic
Marketing, he was in charge of driving webMethods agenda through media and analyst
relations. Gary has a Masters degree in Computer Engineering from the University of
Toronto.
Beth Gold-Bernstein
VP, ebizQ
Beth Gold-Bernstein is a recognized expert in integration
technologies and SOA. She has over 20 years experience
working with fnancial institutions, retail chains and
manufacturers on planning, designing and implementing
large scale distributed systems and enterprise
architectures. She has developed training programs for
business analysts, architects, and implementers, and speaks nationally and internationally
at conferences and seminars. Beth is the author of Enterprise Integration: the Essential
Guide to Integrated Solutions, published in July 2004 by Addison Wesley. Her frst book,
Designing Enterprise Client/Server Systems, was published by Prentice Hall in August
1997. Beth holds an M.S. in Computer Information Systems from Bentley College.