Increase the Flexibility of MySQL-based SOA Frameworks with a Data Access Layer

Mike Frost
Product Manager DataDirect Technologies

DataDirect Technologies Overview
• #1 independent supplier of data access technologies • Any application, any platform, any data source • Standards leader • Leading the market in addressing changing data access requirements • A rich history, strong backing:
• 20 years in business • Formerly Intersolv, Merant, NEON • Independent operating company of Progress Software (Nasdaq: PRGS)
MySQL Oracle DB2 SQL Server Sybase Informix Teradata XML Adabas IMS VSAM CICS IAM IDMS EDI ODBC JDBC ADO.NET XQuery Web Services

• MySQL partner since 2007

Agenda
• Where are we today?
• SOA • Data access

• Data access problems in SOA environments • Solving the problems • What does the future hold?

Agenda
• Where are we today?
• SOA • Data access

• Data access problems in SOA environments • Solving the problems • What does the future hold?

The current SOA landscape
• Many corporate IT organizations are evaluating, planning, or implementing SOA environments
• Common framework for development of new applications • Leverages standards such as XML, SOAP, etc.

• A collection of business goals drives this decision
• • • • • • Control IT costs Increase overall business nimbleness and flexibility Develop and deploy new applications faster Reuse business logic common to multiple applications Optimize use of hardware / software investments Build bridges between heterogeneous environments

The SOA vision
11010100 1101010010 1010101010 0010100101 0101010001 11010100101 01010101000 10100101010 10100010011 10101010 10100010 10010101 01010001

11000000111 00111100 0011110000 1010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011 1101010 1001010101 00001111 101010100000101011010111101100010100001001100101011101010010100101011110010010101011001010001101010100010101101110101010000010 1101010 0011110010 1101010010101 101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100 1101010 0101010 00101010 101101011110110001010000100110010101110101001010010101111001001010101100101000110101010001010110111010101000001010110101111011 0101010 1010 0101010001010 0011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011 0101010 1010100 1 000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010 1010100 0101010101000 001010000100110010101110101001010010101111001001010101100101000110101010001010110111010101000001010110101111011000101000010011 1010100 0101001 111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000 0101001 1001111000000 0101001 0101010 0101010 1110100011101 0101010 1000100 1000100 0101101010011 1000100 1111000 1111000 1010101000101 1111000 0001111 0001111 0101000110101 0001111 0010101 0010101 0011010101110 0010101 01 01 1001101010010 01 1011101001001 11001001101

The current data access landscape
• Defined interfaces
• • • • • OCI, ct-lib, etc. ODBC JDBC ADO.NET OLE DB

• Interfaces have evolved – ground breaking changes are not functional capabilities • All of these interfaces do roughly do the same thing
• Provide APIs to connect to data sources, issue queries, return data, etc. • Reuse legacy business knowledge

Data access in traditional application architectures

MySQL

Data access in the real world

MySQL

Oracle

MS SQL Server

DB2 / z/OS business logic

Agenda
• Where are we today?
• SOA • Data access

• Data access problems in SOA environments • Best practices for data access in SOA environments • What does the future hold?

Data access in a typical SOA architecture

MySQL

What problems does this cause for SOA environments?
• Performance / scalability requirements underachieved
• “Best in class”/reusable services initially designed for 50 users might be deployed in a SOA environment with 500 users • Lack of understanding of the mechanics of data access

• Deployment slowdowns
• A small deployment headache quickly becomes a huge snafu as the number of services increases • Services typically built in a DBMS-centric fashion, limiting reusability

• Higher system TCO
• Reducing heterogeneity is time-consuming and difficult • Consolidation works only if identical or better performance can be achieved

Why does this happen?
• Starting with what is easiest / most comfortable
• What works for MySQL may not be right for other data sources (and vice-versa) • “SOA experts” provide SOA guidelines – not data architects

• Data access is usually an afterthought or checklist item • “SOA-enablement” = expose applications as web services • SOA presents a very different approach….
• Loosely coupled • Stateless • Synchronous / Asynchronous • Message based • XML data interchange • “RPC” model

…over the traditional data access model
• • • • • • Tightly coupled Complex State Machine Mostly synchronous Connection based Well defined API Relational model driven
• SELECT then Fetch model

Agenda
• Where are we today?
• SOA • Data access

• Data access problems in SOA environments • Solving the problems • What does the future hold?

How can these problems be solved?
Best Practices for data access in a SOA environment: 3. Decouple business logic and data access logic 5. Optimize the use of heterogeneous environments 7. Addressing persistent data access performance problems 9. Streamline data access with data integration techniques

1. Decouple data access logic from business logic
• Can be done today with any standards-based data access
• Use ODBC, JDBC, ADO.NET, or any other API • Multiple technologies exist to facilitate this architecture today

• Leverage and focus skills of existing developers
• Allows application developers to focus on application development • Leaves data access in the hands of the data access experts • Applications built faster and better

• Selection of the right database middleware is essential for success
• Build all applications with a strong foundation in data access • “A chain is only as strong as its weakest link”

Consider changing this approach…

Invoicing Application

Shipping Application

Contracts Application

Data Access layer

MySQL

…to this approach: a Data Access Layer

Invoicing Application

Shipping Application

Contracts Application

Data Access Layer
ODBC, JDBC, ADO.NET, etc.

MySQL

Key to building a successful Data Access Layer High quality database middleware
• Reliability • Consistency across data sources • High performance & scalability • True conformance to standards • Robust feature and functionality support

What about SOA environments?

MySQL

From Data Access Layer to Data Access Service
• It is possible to “SOA-enable” the data access layer
• In a SOA environment, the data access layer becomes a data access service • Services can now access data presented as a web service

• Avoid accessing data directly inside every service
• Forces business logic experts to become data access experts • Results in extremely complicated deployment scenarios • Reduces scalability and performance

• One of the reasons to adopt SOA is to share “best in class” services across a set of applications
• Accessing data in an optimized manner IS a “best in class” service

• A data access service is “best in class”, scalable, easy to deploy, and reusable

2. Optimize the use of heterogeneous environments
• Tendency to reduce complexity by standardizing on as few environments/data sources as we can imagine
• Typically on those environments/data sources we know best or offer the lowest TCO

• Consolidation is often seen as a way to cut IT costs
• But many IT flexibility problems start because of unforeseen narrowness in the consolidated architecture • These problems waste many of the cost savings of consolidation

• Optimizing can help minimize consolidation pain
• Reliability of new, consolidated architecture • Difficulty achieving identical or better performance • Inconsistency of implementation due to slight differences in data source capabilities • Higher, rather than lower, system TCO

Don’t reduce complexity just for the sake of doing so
Invoicing Application Shipping Application Contracts Application

Data Access Layer
OCI

Oracle

SOA environments can handle heterogeneity if data access is decoupled

MySQL

Oracle

MS SQL Server

DB2 / z/OS business logic

Optimization vs. Consolidation?
• First question – does the existing system work? • If so, first strategy is to assume the SOA principle of ‘reuse’ should apply – in particular if the system is complex
• Migrating and consolidating a complex working system onto an alternative system will cost orders of magnitude more than you think it will • Most “unsolvable” problems will be encountered only after you are in the details or are at deployment time

• If consolidation seems to be the answer, assume that achieving identical performance is the first goal and benchmark* extensively before agreeing to consolidate

3. Addressing persistent data access performance problems
The SOA Data Access Service Lifecycle: • Design Database Schema (tables, partitions, indexes, etc) • Design architecture of the application framework • Design individual applications/services • Build/Test/Deploy individual applications/services • Load Test • Panic: performance is not acceptable – how do I fix it?

Data access code must be written well to avoid performance bottlenecks
• Standards-based APIs provide numerous ways to do the exact same thing • Documentation and coding guidelines that address performance are difficult to find
• Even if found, they may be too general to be helpful

• Bad coding techniques account for many of the data access performance problems people encounter • Education is the key to avoiding some of the most common, yet avoidable blunders
• Simple techniques can improve performance by > 100%

Example: Calling a Stored Procedure
• Examine every API call … example below performs well over 100% faster “{call getMikesBeerInventory ( ‘Hellbender Barleywine’, ‘Pisgah Brewing’, 2008)}” or “{call getMikesBeerInventory (?, ?, ?)}”

Understand the lifecycle of solving data access performance problems
• The “bottleneck” will start in the database
• Design an efficient schema and tune the database

• The “bottleneck” will move to the application or service
• Understand the performance issues surrounding database middleware and then design/tune performanceaware data access code

• The “bottleneck” will shift to your data access middleware
• Choose database middleware that can be tuned with runtime tuning options • Adjust those options to match your application and environment

How database middleware works conceptually

What’s really going on?

Database middleware can make a difference in terms of throughput

Database middleware can determine how well an application or service will scale

4. Streamline data access with data integration techniques
• Most SOA environments begin with simple services doing simple things
• Looking up an order using an order id

• As the environment expands and the comfort level increases, the complexity of services increases
• Retrieve an incoming EDI order, validate client information in internal system and retrieve customer number, place order in another internal system using customer number, generate and send receipt back to client via EDI • Now “accessing” several data sources in a single service, an EDI feed, a data source that stores the client information, a data source where the order is placed, and then another EDI request to send a response

A typical SOA-based data integration problem
The user submits a request for a report about their stock holdings The web server needs to fetch user’s personal data, stock holdings and live stock data to compile a report to send back to the user

A public service offers live (delayed) stock prices

Some obvious pieces …

MySQL

Solve complex data integration problems with XQuery

XQJ + XQuery Implementation

MySQL

Advanced services require multiple operations to access data
• The overhead involved to sequentially process data inside services is extensive
• At some point, data is usually marshaled to/from an XML format • Data must be transformed from one data model (XML for example) to another (relational for example) • Code must be written to do these transformations of data • Multiple invocations of data access services are needed to fulfill the request

• Thinking differently, it is possible to design to integrate the data using XQuery instead of sequentially access the different data stores

Why integrate data with XQuery?
• Simplifies coding • Encapsulates business logic and data integration logic into a single step • Improves performance
• No marshalling/unmarshalling of data • Optimized access to data • Optimized joining/integration of data

Agenda
• Where are we today?
• SOA • Data access

• Data access problems in SOA environments • Solving the problems • What does the future hold?

What does the future hold for data access in SOA environments?
• Emerging standards will transform data access services into robust data application services (DAS) • Organizations will begin to regard implementation of a DAS as key to their IT infrastructure • Database vendors and third party ISVs will begin to compete at offering a complete DAS solution
• Some are already offer solutions that are just a layer on top of existing database middleware components (some low-quality) • Since DAS represents a new abstraction layer, it makes more sense to optimize the connectivity between the DAS and backend data sources

• High quality DAS implementations will allow organizations to maximize their SOA investment

What sorts of standards?
• SDO (Service Data Objects)
• Unified and consistent data access to heterogeneous data sources • Simplified programming model for application programmers • Enables SOA-enabled applications and services to work consistently across heterogeneous data sources

• SDO is an emerging standard that can form the foundation for a DAS solution
• Supported by IBM, BEA, Oracle, SAP, Sybase and... • Version 2 of the specification has been released with implementations in C++, Java, Perl, .NET, and… • Large standards group run through OASIS

• Other standards and specifications

The SOA vision… combined with DAS
11010100 1101010010 1010101010 0010100101 0101010001 11010100101 01010101000 10100101010 10100010011 10101010 10100010 10010101 01010001

11000000111 00111100 0011110000 1010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011 1101010 1001010101 00001111 101010100000101011010111101100010100001001100101011101010010100101011110010010101011001010001101010100010101101110101010000010 1101010 0011110010 1101010010101 101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100 1101010 0101010 00101010 101101011110110001010000100110010101110101001010010101111001001010101100101000110101010001010110111010101000001010110101111011 0101010 1010 0101010001010 0011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011 0101010 1010100 1 000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010 1010100 0101010101000 001010000100110010101110101001010010101111001001010101100101000110101010001010110111010101000001010110101111011000101000010011 1010100 0101001 111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000 0101001 1001111000000 0101001 0101010 0101010 1110100011101 0101010 1000100 1000100 0101101010011 1000100 1111000 1111000 1010101000101 1111000 0001111 0001111 0101000110101 0001111 0010101 0010101 0011010101110 0010101 01 01 1001101010010 01 1011101001001 11001001101

What should you do?
• Make data access a first class citizen of any SOA environment • Don’t wait for “magic beans” to build your data access environment the right way - apply the Best Practices explained here today
1. Decouple business logic and data access logic 3. Optimize the use of heterogeneous environments 5. Addressing persistent data access performance problems 7. Streamline data access with data integration techniques

• Keep an eye on emerging standards that can form the foundation of a complete data services solution

Questions?

Sign up to vote on this title
UsefulNot useful