Internet and Intranet Protocols and Applications

Lecture 14
Introduction to MQSeries And Asynchronous Messaging
May 1, 2002
Joseph Conron Computer Science Department New York University jconron@cs.nyu.edu

What is MQSeries?
A middleware product that implements a messaging and queuing framework. Middleware - an intermediate software component that bridges dissimilar computing environments.
Unix, MVS, OS/400 Tandem, VMS, NT, etc. SNA, NetBios, TCP/IP Cobol, C, JAVA

Messaging and Queueing

Messaging - programs communicate by sending data in messages rather than by calling each other directly. Queuing - messages are put on queues in storage, eliminating the need for programs to be logically connected. A messaging and queuing framework is inherently ASYNCHRONOUS!

Asynchronous vs. Synchronous Communications

Synchronous: App sends request, then blocks until request is processed.
Requires service available at EXACTLY same time as client needs service.

Asynchronous: App sends request and checks at some future time if complete.
Service need not be available when client sends request

Synchronous: good/bad
Good Bad

Easy to program Outcome is known immediately Error recovery easier (usually) Better real-time response (usually)

Service must be up and ready requestor blocks, held resources are tied up Usually requires connection-oriented protocol

Asynchronous: good/bad
Good Bad

Requests need not be targeted to specific server. Service need not be available when request is made No blocking, so resources could be freed Could use connectionless protocol

Response times are unpredictable error handling usually more complex Usually requires connection-oriented protocol Harder to design apps?

Messaging vs. Procedure Calls

As programmers, many of us think procedurally, so using procedures is natural extension of how we think. Messages are an abstract concept: harder for us to conceptualize relationship between actions and messages! But Wait!
Something good happens when we use abstractions!

<Soap Box>
We are free to concentrate on the design of the application itself. We are no longer concerned with details of the environment.

Our application is suddenly portable and to some degree, extensible.

</Soap Box>

A Brief History of MQSeries

1992
Systems Strategies (SSI) develops ezBridge, a messaging and queuing product for VMS, Tandem, and Unix IBM announces Networking Blueprint defining three standard APIs for program to program communication: CPI-C, RPC, MQI

1992-3
State Street Bank (Boston) evaluates IBM messaging product (code name Victory) for IBM CICS/ESA and SSIs ezBridge on VMS and Tandem.
State Street Bank would like to announce the wedding of IBM and Systems Strategies!

1993
IBM buys intellectual property rights for ezBridge from SSI

The Brides Wedding Preparation

(What SSI had to do) Implement IBM Channel Protocol over TCP/IP and LU6.2 (more about channels later). Implement the MQI interface functions (MQCONN, MQOPEN, MQPUT, MQGET, MQCOMMIT, MQCLOSE, MQDISC). Implement MQI Error Semantics (failure conditions should look the same on all systems)

MQSeries/ezBridge Integration Objectives

Applications written in C using ezBridge on VMS and Tandem had to exchange messages with applications written in (COBOL?) Using MQI under CICS/ESA. System intercommunication using channel protocol over TCP/IP and LU6.2 (that is, A VMS system had to look to IBM machine just like another IBM system!).

MQSeries Platform Rollout

Initially, IBMs version of MQSeries ran only on mainframe
(CICS/ESA, IMS/ESA, and eventually VSE).

SSI version (called MQSeries Version 1) initially released on:

VMS, Tandem, AS/400, and Unix (SCO, UnixWare).

1994/1995 -IBM releases the first three distributed platforms:

AIX, OS/2, and AS/400. The AIX version becomes the reference port.

Over time, IBM replaced SSI versions with ports of its reference system. Today - MQSeries runs on over 35 platforms

How Messaging & Queuing Works

Programs communicate by putting messages on queues. Here, program A puts a message on Queue1, which is read by program B.

Note: A and B need not be on the same machine!

How Messaging & Queuing Works (2)

Communication can be oneway or two-way. Here, A sends to B on Queue1, and B responds to A on Queue2

Messaging and Queuing Characteristics

Three key facts about Messaging and Queuing differentiate it from other communication styles:
1) Communicating programs can run at different times. 2) There are no constraints on application structure. 3) Programs are insulated from environmental differences.

Lets examine each of these characteristics in more detail.

Applications Can Run at Different Times

Either program can be unavailable

Key Concept:
message queue exists independently from programs that use them!

No Constraints on Application Structure

There can be a one to many relationship between applications Or a many to one relationship between applications

Applications Shielded from Environmental Differences

Dont care what OS is used. Dont care what language theyre written in Dont care what the underlying communication protocol is used.

Applications Shielded from Environmental Differences

Applications
Programmatic API

Queue Manager Queue Manager

Message Queue

Communications using Message Channels

MQSeries Objects: Queue Manager

Queue Manager
Controls access to queues:
administration (create, delete, etc) usage (Put, Get)

serves as transaction (syncpoint) coordinator for all queue operations. Accessed through the Message Queue Interface (MQI) Queue Managers have names (identities) that are UNIQUE in a network (like host names).

MQSeries Objects: Queues

MQSeries defines four types of queues. A queue instance is fully qualified by its queue manager and queue name.
Local Queue - an actual queue for which storage is allocated.

Remote Queue - a definition of a queue on a different queue manager (acts somewhat like a pointer)
Alias Queue - another name for a local or remote queue. Typically used to switch queue destinations without modifying program code. Model Queue - a template whose properties are copied when creating a new dynamic local queue ( create queue xxx like queue yyy).

MQSeries Queues: Properties

Maximum Message Size Maximum Queue Depth High/Low Factors Enable/Disable Put or Get Persistent/Not Persistent

MQSeries Queues: Events and Triggering

Local queues can generate events (messages) under certain conditions (like queue full). These event messages can be used to trigger the execution of a program. These events are called trigger messages. The queue on which they are put is called an Initiation Queue.

MQSeries Objects: Processes

Process defines an application to an MQSeries queue manager. A process definition object is used for defining applications to be started by a trigger monitor. A trigger monitor is a program that listens on an initiation queue and executes commands named in Process definitions. Triggering is useful when you dont want to deploy longrunning programs.

MQSeries Objects: Message Channels

provide a communication path between two queue managers on the same, or different, platforms. A message channel can transmit messages in one direction only. If two-way communication is required between two queue managers, two message channels are required.

Question: why make message channels one-way?

MQSeries Objects: Message Channels(2)

Types of message channels:
Sender - initiates connection to Receiver Server - Accepts request to start from requester, then becomes Sender Receiver - Passive; waits for initiation sequence form Sender Requester - Active at start, then becomes Receiver Cluster-sender (used amongst Cluster Queue Managers) Cluster-receiver (ditto)

MQSeries: Message Channel Concepts

The Sender side of the session is the transaction coordinator. Message channels implement a protocol that includes a commitment protocol. Channels recover from failure by agreement: they must agree on the last committed unit of work [would this be harder if channels were bi-directional??] Message Chanels are implemented by programs called Message Channel Agents (MCA)

How messages move across channels

(1) Application puts message on transmission queue (4) Message is available on local queue for applications

(2) Sender MCA gets message and sends to partner MCA

(3) Receiver MCA puts message on target queue

MQSeries: Assured Delivery

If queues are persistent, and message is persistent, then MCAs will eventually deliver the message to the target queue, and target application will get it!

MCAs have recoverable state - theyre STATEFUL - and a connection-oriented protocol. Message is not removed from xmit queue until partner MCA confirms placement on target queue

MQSeries Objects: Messages

Messages consist of:
Header (MQMD)
Used by Queue Manager and application to handling properties of the message

User Data
The application-to-application data (payload) transparent to MQSeries

MQSeries Messages: Properties

Destination Queue Reply Queue name Time to live (expiry) Format Correlation ID Persistence Report options

MQSeries Messages: Properties (2)

Messages can be individually designated persistent or non-persistent (persistent messages are logged to enable recovery) Correlation ID - select which message to get from queue Segmented Messages - allows ending of VERY LARGE messages (> 100 MB)

MQSeries Messages: message types

Request - used by one program to ask another program for something (usually data). A request message needs a reply. Reply - used in response to a request message. A one-way message, as you would expect, doesnt need a reply, though it can carry data. A report message is used when something unexpected occurs, or to give extra information like:
message delivered to target queue message taken by application message not deliverable message exceeded time-to-live

MQSeries Messages
Units of Work and Transactions
Messages are added and removed from queues in Units of Work The smallest Unit of Work is one message. Units of work are atomic. When an app reads a message from a queue, a message appears to have been removed, but in fact, it is still in storage until the app commits the unit of work.

MQSeries: Transaction Support

Unit of recovery - a piece of work that changes data from one point of consistency to another.

Syncpoint - A point of consistency (also called a or commit point). It is a moment at which all the recoverable data that an application program accesses is consistent.

MQSeries: Transaction Support (2)

Applications are responsible for delimiting the beginning and end of a transaction. How can messaging be coordinated with a data base update?

MQSeries is XA compliant and can operate with other XA compliant systems as either a transaction manager (coordinator) or resource manager (particpant).
Some examples: Sybase, DB2, Oracle.

MQSeries: Logging and Recovery

All operations that affect the state of the Queue Manager and its objects are logged to a log file.

What is state?
Object definitions (queue manager, queues, processes, channels, etc) Queue content (messages) What about message channel state?

Message channel states are logged separately by each channel

MQSeries: Logging and Recovery (2)

Two forms of Logging
Circular log records are written sequentially across several files, then wrap back to the first file. Linear - log records are written sequentially across files. New files are allocated as current files fill. No automatic reuse of file space!

Problem: Length (in time) of longest running transaction vs amount of writes to log determines size of log needed.

MQSeries: Logging and Recovery (3)

Observations:
Circular logging is easy to manage, but is fatal if log is damaged (hard to backup circular logs!). Linear logging is hard to maintain but provides for archiving of previous logs (still a problem if current log is damaged).

MQI: The MQSeries Programming Interface

MQCONN MQDISC MQOPEN MQCLOSE MQPUT Connect to queue manager Disconnect from queue manager Open queue Close queue Put message

MQGET

Get message

MQI: The MQSeries Programming Interface (2)

MQPUT1
MQBEGIN MQCMIT MQBACK MQINQ MQSET

Put one message (implied open/put/close)

Begin unit of work Commit Back out Inquire about queue attributes Set queue attributes

MQSeries: Language Support

C C++ Cobol JAVA (native and JMS) PL/I System 390 Assembler TAL (Tandem) Visual Basic

For More Information: http://www-4.ibm.com/software/ts/mqseries/