Session Number: M17

Session Number: M17
WebSphere Message Broker Best Practice: How to develop MessageFlows with a reusable framework for error handling, logging and routing
Jan-Philipp Schiller (IT-Specialist, ISSW Germany, jan.schiller@de.ibm.com)
2010 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM.
IBM Software Group | WebSphere Software
1 2 3 4 5 6 7 8 9
Motivation and preconditions SOA Reference Architecture / Patterns Why a framework? Framework architecture, services and components Demo Lessons learned Outlook Asset key facts Questions
Session M17: WMB Best Practice: How to develop MessageFlows with a reusable framework for error handling, logging and routing
2010 IBM Corporation
Objectives
Show a best practice for using WebSphere Message Broker (WMB) within a reusable framework Talk about WebSphere Message Broker / Enterprise Service Bus (ESB) related patterns Demonstrate the flexibility of WebSphere Message Broker Motivate to design, develop and use reusable MessageFlows within WMB e.g. via MessageFlow templates, subflows, WMB Patterns Authoring feature and Patterns Explorer Live demonstration of some framework functions and features Provide some information about the IBM asset: WebSphere Message Broker Service Oriented Framework (WMBSOF)
1 2 3 4 5 6 7 8 9
Motivation and preconditions

Framework was designed in a customer engagement at first and has been steadily developed further Many customer scenarios show: Isolated MessageFlows which grew in complexity in course of time and are difficult to maintain Use of standard WMB product and features Different developers with various skill levels (WMB and WMQ) Reduce development time Platform reuse for future projects Change framework configuration during runtime of WebSphere Message Broker Provide a Starter Pack for new WMB customers Should be usable for the most WMB scenarios
1 2 3 4 5 6 7 8 9
IBM SOA Reference Architecture

Model Assemble Deploy Manage
Business Innovation & Optimization Services

Facilitates better decision-making with real-time business information
Development Services
Enables collaboration between people, processes & information
Orchestrate and automate business processes
Manages diverse data and content in a unified manner
Connect with trading partners
Build on a robust, scaleable, and secure services environment
Facilitates interactions with existing information and application assets
Infrastructure Services
Optimizes throughput, availability and performance
Apps & Info Assets
Integrated environment for design and creation of solution assets
Facilitates communication
ESB
between services
Partner Services
Business App Services
Access Services
Manage and secure services, applications & resources
IT Service Management
Interaction Services
Process Services
Patterns (I)
Development Services
Security Management
Application Services
Interaction Partner Process Business App Information Access
ESB
Registry
Enterprise Service Bus

Mediation Flows Mediation Flows Message Models Message Models Communication Protocols Communication Protocols
Mediation Patterns Meta models Interaction Patterns
Management Services
Patterns (II): Pattern categories

Patterns can be seperated into different pattern categories
Service virtualization (e.g. loose coupling between services, security, logging, monitoring, charging etc.)
Service enablement (e.g. provides access to services like integrating legacy applications into the new world) Gateway (e.g. routing, authentication, protocol switch, audit, etc.) Message-based integration (e.g. message routing, message translator, message aggregator, message splitter etc.) File processing (e.g. file record processing, file record batching, file record distributing etc.) Event-driven integration (e.g. react to events, extract events, filter events, distribute events etc.)
Enterprise Connectivity Patterns: Implementing integration solutions with IBM's Enterprise Service Bus products [http://www.ibm.com/developerworks/library/ws-enterpriseconnectivitypatterns/]
9 Session M17: WMB Best Practice: How to develop MessageFlows with a reusable framework for error handling, logging and routing 2010 IBM Corporation
Patterns (III): Some communication / mediation patterns
Request / Response
Request / Multi Response
Event Propagation
Protocol Switch
Transform
Enrich
Route
Distribute
Monitor
Correlate
+
Canonical Adapter
+
Gateway
Transform Log - Route
IBM Systems Journal Vol. 44, No. 4, 2005 - Service-Oriented Architecture [http://researchweb.watson.ibm.com/journal/sj44-4.html]
10
WMB 7 patterns support

Since version 7.0 (GA) WMB now provides common patterns through the Patterns Explorer These patterns provide templates for developers to create services and components based on common patterns Through the Patterns Explorer the corresponding Message Broker artifacts like MessageFlows, ESQL files, MessageSets are created For more information on using patterns in WMB and to create your own patterns with the new Patterns Authoring feature (since WMB 7.0.0.1) see sessions:
M23: WebSphere Message Broker Patterns ML4: WebSphere Message Broker Hands-on Lab
11
1 2 3 4 5 6 7 8 9
Motivation and preconditions SOA Reference Architecture / ESB related patterns Why a framework? Framework architecture, services and components Demo Lessons learned Outlook Asset key facts Questions
12
Why a framework?
Until version 7.0 WMB did not provide common services like: Error handling Routing Logging Restarting Notification ... Such services are needed in nearly every customer engagement Should be easy to extend and to customize (individual customer requirements) Make use of identified patterns and create development templates (subflows) Reuse of components Reduce effort and development time Developers should concentrate on the process flow (e.g. mapping, transformation, sequencing, aggregation etc.)
13
High level view of a generic framework application

WMB framework part Framework GUI part
Status Monitoring
Interface check
Resource check
Configuration
Dispatcher (Framework / Individual)
Process Process (individual) Process (individual) (individual)
Process Process Deliver (individual) (individual) (Framework individual)
Logging Error handling Routing Notification Queue
Logging Error handling Notification Restart Queue
Logging Error handling Notification Restart
Maintenance
14
Functional overview of the framework

Service driven framework Separation into the following services and components (in brackets)
Dispatcher Processing Delivery (Value Mapping) (Error handling) Logging Notification Restart Maintenance
Toolbox: Configurable MessageFlows / subflows as templates Configuration database (key component) Administration GUI
15
1 2 3 4 5 6 7 8 9
16
Framework architecture
Internal Systems Framework Administration GUI
Configuration
External Systems
Adapter Siebel SAP File ...
Monitoring Restart Analyze
Status propagation
Configuration DB
Routing Validation Scheduling Communication Value Mapping
Dispatching
Individual
Processing
Individual
Delivering
Individual
Value Mapping Logging, Restarting, Notification
Log Archive
Error handling
17
Service: Dispatcher (I)

Request - Protocol Switch - Enrich - Log - Route
WebSphere MQ HTTP/S Flat File (via FTP) SAP Adapter Siebel Adapter SQL WebSphere TX Extendable (SOAP, JMS, etc.)
Error handling subflow
Subflow for File Dispatcher
18
Receiving messages from internal and external systems via
Service: Dispatcher (II)

Central entry point for message processing Identifies sender, message type, receiver and determines process and delivery flow Determine routing information for further processing in individual part of the flow or set directly Routing information is maintained in the configuration database via GUI. The dispatcher service extracts it out of the configuration database only service that accesses the configuration database Stores routing and framework data in the MQRFH2 header of the message Dispatching service stores payload via logging service into log-event table Sends message to next process step Individual Part In case of an error: logs the erroneous message, Dispatcher Subflow its metadata as well as the error reason via error handling component for a later restart
File Input Subflow
Example of Dispatcher Flow

19
Service: Dispatcher (III) Rules for routing

Routing defines the path a message takes through the MessageFlows. A path can be assembled dynamically by changing the settings in the database via the GUI.
Sender
MsgType
Rule
Receiver
Process From SAP (Example) Sender is extracted from SAP control header Receiver is extracted from SAP control header MsgType is extracted from SAP control header
Delivery service From external (Example) Sender is extracted from message, URL, Receiver is extracted from message, URL, ... MsgType is extracted from message, URL,
20
Service: Dispatcher (IV) Multiple receivers

Common use case: deliver transformed message to several target systems (fan-out)
Must have at least the same sender and message type Different receivers are stored in <usr> section of the MQRFH2 header Subsequent Processing MessageFlow (mapping) will only run once Transformed message is sent out to the configured multiple receivers
Sender
MsgType
Rule
Receiver
Process
Delivery service
21
Service: Dispatcher (V)

1. 2. Set properties Create routing info
SAP Error handling MQ
SAP
MQ
HTTP/S
Siebel
File
SQL
Other protocols
MQ
HTTP/S
HTTP/S
Siebel
File
DB
...
(WTX)
1. Individual
Selection criteria: Selection criteria: Message Type Message Type Sender Sender Receiver Receiver MQ MQ
Dynamic ports
2.
MQ
Routing tables
Logging
MQ
22
Service: Processing (I)

Responsible for message format + + transformation and process logic (sequencing, collecting, mapping, ) Transform / Enrich Log Route Is not part of the framework. The developer has to develop the process flows MQ entry point (MQ InputNode) Uses common framework services and components Each process flow is decoupled by MQ queues at the start and end of the processing Logging is done via the logging service generates a log message Sends the message to the next process step (e.g. Delivery) Error case: Logs erroneous message, its metadata as well as the error reason via the error handling component for a possible later restart (restart service)
23
Service: Processing (II)

Error handling
MQ
MQ
Process frame
MQ
MQ
Try Process logic MQ

MQ
MQ
Logging tables
Logging
MQ
Value mapping
Mapping tables
24
Service: Delivery (I)

+ + MQ entry point Sends message payload to the assigned adapter Log Distribute Protocol Switch according to the metadata in the MQRFH2 header Uses error handling, notification and logging service In case of an error logs erroneous message, its metadata as well as the error reason via the error handling component for a later restart Sends the message to the target application / system in the correct protocol
MQ Input Subflow HTTP Delivery Subflow
Example of Delivery Flow
Sub Flow for Delivery
25
Service: Delivery (II)
Dedicated
Siebel
SAP
MQ
File
HTTP/S
MQ
ports
(WTX) MQ
MQ
Dynamic ports
Port dispatcher
MQ
MQ
Logging
MQ
Error handling
MQ
26
Example 1: Send MQ message to a file
Dispatch
Process
Deliver
1 MQ
5 MQ 3 4
Routing tables
8 MQ 7
FTP 13
11 10 12
Logging tables
Event tables
Message is put on a MQ queue Message is delivered into a file system Message is logged three times
Notification
27
Component: Value Mapping

Translates codes like language codes, partner or material numbers from external to internal representation Simple key-value pairs Is called by the processing service Data is stored in the database and is maintained via the GUI Access to value mapping data via encapsulated procedures
Enrich
DE EN ES
GetApplicationXRef() GetApplicationXRef() GetApplicationXRef()
German English Spanish
Example Value Map
28
Component: Error handling

Logs erroneous message (payload), its metadata + as well as the error reason for a possible restart as XML Log Event propagation in the log-event database (XML datatype) Receives data from dispatching, processing or delivery service Is part of every MessageFlow as a subflow Delivers message also to a defined error queue Triggers notification service for registered users (Email) Uses logging service
Error handling subflow

Service: Logging
Responsible for storing message payload in the database for analysis, status monitoring, auditing, tracking, restart or correlation with messages at a later time MQ input Uses common error handling component and notification service Store message content in the log-event table Can log payload and header for auditing reasons Different log levels (no logging, framework header logging, payload + header Logged data (payload and header) is accessible through the GUI
Log
Logging Flow
Service: Restart
+ + Looks for erroneous messages in the log-event table and according to the Correlate Enrich Route configuration it tries to restart the message If required runs one or more restarts by putting payload back into the service input queues Stops resending after n times. N is a configurable parameter in the database for this specific sender processing receiver combination Restart is only available for services which have a MQInput node Manual restart can be done via the GUI
Restart Flow
Service: Notification
Sends Email in case of an error or + + successful processing to registered users Event propagation Enrich - Protocol Switch Email can contain payload, header information and exception list MQ input Uses configurable templates for Email Uses different event types for sending notifications Templates and mailing contacts can be maintained via the GUI
Notification Flow
Service: Maintenance
Provides maintenance functionality:
Archive / empty queues Archive / empty files system folders Archive / empty database tables
Central MessageFlow which regularly checks for maintenance jobs Completely configurable through the GUI Prevents symptoms like full file systems, queues and database tables Example: Archive and delete all records from the log-event table which are older than 60 days
33
Toolbox
Template flows are provided for:
File Delivery HTTP/s Delivery MQ Delivery Processing File Dispatching HTTP/S Dispatching MQ Dispatching SQL Dispatching ...
Set of subflows: Reduced development time Developer can use templates and subflows via drag & drop for flow development Efficient code reuse
34
Framework Database
The framework uses the DB2 9.5 XML feature (pureXML) to store the payload and header information in a XML data type (query with XPATH) Database contains
Tables to get the routing information Tables to log the incoming events and payload information Tables to maintain the notification events Table for maintenance ...
Whole framework is configurable through the database
35
Administration GUI
Single point of information Status monitoring Intuitive administration and configuration of the framework Role-based user management Part of the asset GUI technology
Runtime: WebSphere Application Server 7.0 (WAS), Java 6 Frontend: Java Server Faces (JSF) Java Persistence API (JPA) IDE: Rational Application Developer 7.5 Delivered as an Enterprise Application (EAR file)
Currently only tested with WebSphere Application Server 7

1 2 3 4 5 6 7 8 9
37
High level view of a generic framework application

WMB framework part Framework GUI part
Status Monitoring
Interface check
Resource check
Configuration
Dispatcher (Framework / Individual)
Process Process (individual) Process (individual) (individual)
Process Process Deliver (individual) (individual) (Framework individual)
Logging Error handling Routing Notification Queue
Logging Error handling Notification Restart Queue
Logging Error handling Notification Restart
Maintenance
38
Demo
39
1 2 3 4 5 6 7 8 9
40
Lessons learned
Development of such a framework created additional effort but upcoming interfaces / projects will be developed much faster Rapid flow development Easy to extend More and more features get implemented Little complexity of flows Even developers with little WMB and WMQ skill can use this framework Framework is designed for the most common WMB use cases Deployed on several customer systems (WMB 6.1/7)
41
1 2 3 4 5 6 7 8 9
42
Outlook
Evaluate support of different application servers, databases and browsers (only tested with WAS 7, DB2 9.5 and Mozilla Firefox 3.x) Reuse this framework in future customer projects Provide framework as patterns through Patterns Authoring feature Integrate upcoming WMB features Can easily extended to use WebSphere Business Monitor for monitoring
43
How does the asset integrate with WMB patterns?

Currently the framework is provided as: MessageFlow templates, subflows and generic ESQL procedures etc. Framework (WMB part) is delivered as Project Interchange file Future goal is to integrate the framework into the WMB Patterns Explorer using the new Patterns Authoring feature introduced in WMB 7.0.0.1 Distribute framework as pattern plugin (export from Patterns Authoring) Delivery and configuration of the framework in a WMB standard way using WMB patterns Patterns in WMB are helping users in configuring and using the asset
44
1 2 3 4 5 6 7 8 9
45
Asset Key Facts

Asset name: WMB Service Oriented Framework (WMBSOF) Starter package for new WebSphere Message Broker customers
Admistration GUI for easy configuration and status monitoring (WAS 7 EAR) Configuration database (from DB2 9.5 onwards) Development and runtime framework (WMB 6.1/7 MessageFlows) Easy installation via scripts (setup in a short time frame) Development and installation documentation Demo-Application
IBM ISSW Service Asset

Will be distributed only via an ISSW service contract engagement AS-IS asset (no official IBM support or guarantee; it is not a product!) GUI and framework can be branded for each customer
46
Asset Contacts
ISSW Germany sales contact:
Sonja Steiner (ISSW Germany Service Sales), sonja.steiner@de.ibm.com
Technical contacts and developers of the WMBSOF asset:

Sabine Busch (ISSW Germany), buschs@de.ibm.com Jan-Philipp Schiller (ISSW Germany), jan.schiller@de.ibm.com
47
1 2 3 4 5 6 7 8 9
Motivation and preconditions SOA Reference Architecture / ESB related patterns Why a Framework?! Framework architecture, services and components Demo Lessons learned Outlook Asset key facts Questions
48
Questions
Please remember to complete your session evaluation form
49
Ressources
IBM Systems Journal Vol. 44, No. 4, 2005 - Service-Oriented Architecture http://researchweb.watson.ibm.com/journal/sj44-4.html Enterprise Connectivity Patterns http://www.ibm.com/developerworks/library/ws-enterpriseconnectivitypatterns/ IBM ESB and Connectivity Patterns Wiki https://www.ibm.com/developerworks/wikis/display/esbpatterns/ESB+and+Connectivity +Patterns IBM Patterns for e-business library http://www.ibm.com/developerworks/patterns/library/index.html IBM WebSphere Message Broker product information http://www-01.ibm.com/software/integration/wbimessagebroker/ IBM WebSphere Message Broker Information Center V7 http://publib.boulder.ibm.com/infocenter/wmbhelp/v7r0m0/index.jsp IBM WebSphere Message Broker Library http://www-01.ibm.com/software/integration/wbimessagebroker/library/ IBM DB2 9 Product Information http://www-01.ibm.com/software/data/db2/9/
50

Session Number: M17

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Session Number: M17

Uploaded by

Copyright:

Available Formats

Session Number: M17

IBM Software Group | WebSphere Software

2010 IBM Corporation

IBM Software Group | WebSphere Software

2010 IBM Corporation

IBM Software Group | WebSphere Software

2010 IBM Corporation

IBM Software Group | WebSphere Software

Motivation and preconditions

2010 IBM Corporation

IBM Software Group | WebSphere Software

2010 IBM Corporation

IBM Software Group | WebSphere Software

IBM SOA Reference Architecture

Business Innovation & Optimization Services

Enables collaboration between people, processes & information

Orchestrate and automate business processes

Manages diverse data and content in a unified manner

Connect with trading partners

Build on a robust, scaleable, and secure services environment

Facilitates interactions with existing information and application assets

2010 IBM Corporation

Apps & Info Assets

Integrated environment for design and creation of solution assets

Business App Services

Manage and secure services, applications & resources

IBM Software Group | WebSphere Software

Enterprise Service Bus

IBM Software Group | WebSphere Software

Patterns (II): Pattern categories

IBM Software Group | WebSphere Software

Patterns (III): Some communication / mediation patterns

Request / Multi Response

Transform Log - Route

2010 IBM Corporation

IBM Software Group | WebSphere Software

WMB 7 patterns support

2010 IBM Corporation

IBM Software Group | WebSphere Software

2010 IBM Corporation

IBM Software Group | WebSphere Software

2010 IBM Corporation

IBM Software Group | WebSphere Software

High level view of a generic framework application

Dispatcher (Framework / Individual)

Process Process (individual) Process (individual) (individual)

Process Process Deliver (individual) (individual) (Framework individual)

Logging Error handling Routing Notification Queue

Logging Error handling Notification Restart Queue

Logging Error handling Notification Restart

2010 IBM Corporation

IBM Software Group | WebSphere Software

Functional overview of the framework

2010 IBM Corporation

IBM Software Group | WebSphere Software

2010 IBM Corporation

IBM Software Group | WebSphere Software

Adapter Siebel SAP File ...

Monitoring Restart Analyze

Routing Validation Scheduling Communication Value Mapping

Value Mapping Logging, Restarting, Notification

2010 IBM Corporation