Websphere MQ Data Conversion PDF

IBM Software Group
WebSphere MQ Data Conversion
Blanche Ishida and Karen Stanley

WebSphere MQ Support
WebSphere Support Technical Exchange

IBM Software Group
Agenda
Terms and Definitions
WebSphere MQ Message Header (MQMD)
Customer Scenarios
IBM Technotes
Useful Links
Summary
Questions and Answers
WebSphere Support Technical Exchange 2

IBM Software Group

Character Set
Is a defined set of characters used to represent textual
information (e.g. 0-9, a-z, A-Z, .,;:!?/-)
A Character Set generally supports more than one
language
Code Page
(AKA Code set) A code page can be defined as a table
with a mapping of alphanumeric code and its binary
representation.
Where each character in a character set is assigned a
numerical representation (often used interchangeably
with character set e.g. charset in HTML)

IBM Software Group

A code point is the location of a character within
the code page
CCSID Coded Character Set ID is a unique

number (0-65535) used by IBM to uniquely identify
a Coded Character Set and a Codepage.

IBM Software Group

ASCII : American Standard Code for Information Interchange.
The code used as the basis for most codes apart from
EBCDIC.
EBCDIC : Extended Binary Coded Decimal Information Code
(EBCDIC). It is typically used on z/OS, iSeries and VSE
UNICODE : A code page which defines most of the
characters used by the languages now used throughout the
world. UCS-2, UTF-8, UTF-16 and UTF-32 for encodings.
SBCS Single Byte Character Set. A code page with up to 256
characters which can be represented by a single 8 bit byte.
DBCS Double Byte Character Set. A code page with up to
65536 characters.

IBM Software Group
MQMD
The Message Descriptor is not entirely character data, so
some of it will be converted as character data, some as
numeric, and some is passed unconverted.
Note: MsgId, CorrelId, and Accounting Token are NOT

considered to be character data, and as such WILL NOT be
converted as character data. Many applications put
character data into these fields, but if the message is being
exchanged between ASCII and EBCDIC platforms, and the
application on the other side needs to be able to read these
fields, then it is the responsibility of the application (or a user
exit) to convert these fields. Also the data maybe in
hexadecimal representation for MsgID and CorrelID.

IBM Software Group
WebSphere MQ Message Header

StrucId Structure identifier Priority Message priority
Version Structure version Persistence Message persistence
number MsgId Message identifier
Report Options for report
messages CorrelId Correlation identifier
MsgType Message type BackoutCount Backout counter
Expiry Message lifetime ReplyToQ Name of reply queue
Feedback Feedback or reason ReplyToQMgr Name of reply queue
code
Encoding Numeric encoding of manager
message data UserIdentifier User identifier
CodedCharSetId Character set
identifier of message data AccountingToken Accounting token
Format Format name of
message data

IBM Software Group
WebSphere MQ Message Header cont

ApplIdentityData Application Note: The remaining fields are
data relating to identity ignored if Version is less than
PutApplType Type of MQMD_VERSION_2.
application that put the
message GroupId Group identifier
PutApplName Name of MsgSeqNumber Sequence number
application that put the of logical message within group
message Offset Offset of data in physical
PutDate Date when message
was put in GMT message from start of logical
PutTime Time when message message
was put in GMT MsgFlags Message flags
ApplOriginData Application data OriginalLength Length of original
relating to origin
message

IBM Software Group
MQMD
There are two parts to a MQ message.
There is a Message Descriptor (MQMD), followed by User data.
D4C44040 00000001 00000000 00000008 MD ............

00007530 00000000 00000311 000001F4 ..............4
D4D8E2E3 D9404040 00000000 00000000 MQSTR ........
StrucId : 'MD ' Version : 2

Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 546 CodedCharSetId : 437
Format : 'MQSTR '
Priority : 0 Persistence : 0

IBM Software Group
MQMD : Encoding
Encoding is generally taken to mean the method
that this platform uses to represent numeric data.
There are two general types:
LittleEndian, used by Intel processors (e.g
Windows, Linux on Intel). e.g. the number 437
would be represented (in hex) as XBF01 (we
refer to this as byte swap)
BigEndian. e.g. the number 437 would be
represented (in hex) as X01BF.

IBM Software Group
Encoding scheme by platform

Platform Encoding Scheme
zSeries BigEndian
AIX BigEndian
iSeries BigEndian
HP-UX BigEndian
Windows LittleEndian
SINIX BigEndian
Sun Solaris (on SPARC processors) BigEndian
Sun Solaris (on INTEL processors) LittleEndian
Linux (Intel) LittleEndian
Linux (zSeries) BigEndian
NonStop Kernel (NSK) BigEndian
OVMS Alpha BigEndian
Open VMS VAX BigEndian
Tru64 Unix BigEndian

IBM Software Group
CCSID
The Queue Manager CCSID is important as it
defines the default CCSID for messages being
created using MQPUT and is the default CCSID
which will be used for conversion when messages
are retrieved using MQGET. If the channel option
CONVERT(YES) is set the conversion is attempted
for the receiving Queue Managers CCSID.
To display the queue manager CCSID use the

DIS QMGR command

IBM Software Group
MQMD : CodedCharSetId
Some well known CCSIDs:
CCSID 437 -- this is an ASCII code page, used mainly under

OS/2, DOS, and Microsoft Windows console (OEM) windows.
CCSID 850 -- this is another common ASCII code page, used
mainly on under OS/2, DOS, and Microsoft Windows console
(OEM) windows used on PCs in Europe and some AIX
locales.
CCSID 819 -- this is the ISO 8859-1 standard Western
European ASCII code page. Used by most UNIX locales.
CCSID 500 -- This is an EBCDIC code page, used mainly on
z/OS. It is known as the International codepage.
CCSID 037 -- This is another popular EBCDIC code page,
used on OS/400. It is the US English codepage.

IBM Software Group
MQMD : Format
This indicates to the receiver of a message the format of the
application data in the message.
The queue manager has a number of built-in formats
with names beginning MQ, for example MQFMT_STRING. If these
do not meet your needs, you can define your own formats (user-
defined formats), but you must not use names beginning with MQ for
these. When you create and use your own formats, you must write a
data-conversion exit to support a program getting the message
using MQGMO_CONVERT.
The format name MQFMT_NONE is a special value that indicates
that the nature of the data in the message is undefined. As a
consequence, the queue manager does not attempt conversion
when the message is retrieved from the queue.

IBM Software Group
FORMAT
The queue manager built-in formats all have names beginning with
MQFMT. They are listed and described in the WebSphere MQ
Application Programming Reference under the Format field of the
Message descriptor (MQMD)
Some built-in formats
MQFMT_NONE (default)
MQFMT_STRING (for character data conversion)
MQFMT_DEAD_LETTER_HEADER
MQFMT_IMS
MQFMT_IMS_VAR_STRING
MQFMT_PCF
MQFMT_RF_HEADER
MQFMT_RF_HEADER_2
MQFMT_TRIGGER
MQFMT_XMIT_Q_HEADER

IBM Software Group
Customer Scenario 1
iSeries/Sender VSE/Receiver
WMQ Level 5.3 WMQ Level 2.1.2
OS Level 530 OS Level ???
QMGR CCSID 37 QMGR CCSID 1047
Sender Convert *YES
Problem : Sender channel fails with message AMQ9541

Message . . . . : CCSID supplied for data conversion not supported.
Cause . . . . . : The program ended because, either the source CCSID '&1'
or the target CCSID '&2' is not valid, or is not currently supported.
Recovery . . . : Correct the CCSID that is not valid, or ensure that the
requested CCSID can be supported.

IBM Software Group
Customer Scenario 1 cont

Solution : At the time this issue was submitted the
customer had the following options
Change CONVERT to *NO on the Sender channel
Change VSE QMGR CCSID to 924/37
As of May/2007 CCSID 1047 has now been added,
along with CCSIDs 5346, 5347, 5349, 5350, 5351,
5352, and 5353 to the iSeries server. Reference apar
SE28865 (PTF SI277906 for OS/530 and SI27516 for
OS/540).

IBM Software Group
Customer Scenario 2
iSeries/Sender zOS/Receiver
WMQ Level 5.3.12 WMQ Level 5.3.1
OS Level 530 OS Level ???
Sender Convert *NO
Problem : Enduser indicated that they have an iSeries

communicating with z/OS. The sender channel is going into
RETRYING status and message AMQ9503 (Channel
negotiation failed) is generated when the PNGMQMCHL is
issued.

IBM Software Group
Solution : Found that the receiver/responder

determined it could not handle the Fap level and/or
encoding scheme being negotiated between the
channels.
Work-Around
Implement an intermediate queue manager
iSeries (CCSID 37) z/OS (CCSID 5035)
z/OS (CCSID 5026)

IBM Software Group

Work-Around
Implement server/requester channel pair
iSeries (CCSID 37) z/OS (CCSID 5026)

This configuration will only work as long as the
initiator is the z/OS
WMQ z/OS L3 team did find that this was a
defect and created a usermod, however it was
not verified by the customer as they decided to
continue to function with the intermediate queue
manger in place.

IBM Software Group
What Gets Converted,?

For Channel Communication -- When a channel between two
WebSphere MQ Queue Managers is started, the two Queue Managers
will negotiate which CCSID and encoding protocol they will use for
channel communications.
Reference IBM Technote 7005729 : Data Conversion under

WebSphere MQ

IBM Software Group
Customer Scenario 3
Windows/Sender iSeries/Receiver
OS Level ??? OS Level ???
Problem : Characters |, ] and ! are not being properly

handled when sent from Windows to iSeries.

IBM Software Group

Solution : Change the queue manager on iSeries to
CCSID 37 or a create a channel exit program that will
then handle the conversion.
Reference IBM Technote 1207860 : Conversion

character difference between CCSID 37 and 500

IBM Software Group
Code page 37 - Code page 500 differences

IBM Software Group
Common problem code points

Character CCSID 437 CCSID 37 CCSID 500
! A5 5A 4F
F5 5F BA
A4 4A B0
| F4 4F BB
[ AB BA A4
] BB BB A5
^ 0B B0 5F

IBM Software Group
Customer Scenario 4
Sun Solaris/WMQ Client z/OS /SVRCONN and RCVR
OS Level ??? OS Level ???
Putting Application CCSID 819 QMGR CCSID 500
Must have Client attachment
z/OS / Sender
WMQ Level
Problem: Messages put to z/OS
OS Level ??? queue with a CCSID of 819 are
QMGR CCSID 500 processed properly by the
Getting client (Windows)
Windows / WMQ Client application. However,
WMQ Level messages put by the z/OS
OS Level queue manager with CCSID of
Getting Application CCSID 1252 500 are not converted by getting
client application.

IBM Software Group

Solution : Messages PUT from the z/OS queue
manager or any platform should have MQFMT_STRING
format specified in the message header. Also the
Getting Client application should make sure it has
CCSID 437 specified, along with MQGMO option
Convert.
Reference: WebSphere MQ Application Programming

Reference (SC34-6062)

IBM Software Group
The message is entirely character data

If the message you are passing is entirely character data then it is
very easy to have WebSphere MQ do all of the work for you. In the
application that is doing the MQPUT of the message, you should
set MQMD.Format to MQFMT_STRING. MQFMT_STRING is a
constant that equates to the string value MQSTR . This value in
the Format field will let WebSphere MQ know that the message
consists entirely of character string data.
Reference IBM Technote 7005729 : Data Conversion

under WebSphere MQ

IBM Software Group
Common Procedure to ensure conversion on a

MQGET
In the application doing the MQGET of the message, set MQMD.CodedCharSetId to
0 to pick up the default CCSID of the QMGR. This will ensure that WebSphere MQ
knows the correct CCSID to convert to.
In the application that is doing the MQGET of the message, make sure you specify
MQGMO_CONVERT as one of the MQGMO.Options. This field has many options
that should be either added or bitwise ORed together if you need to specify more
than one.
It is recommended that you always do the data conversion on the MQGET of the
message, since the message may take many hops across different platforms on its
way to its destination, and doing data conversion on each hop is a waste of
resources.
If you specify CONVERT(YES) on the SENDER channel that sends the data to its
final destination, WebSphere MQ will convert the data into the CCSID of the queue
manager on the target system.

IBM Software Group
Customer Scenario 5
iSeries /SDR Tandem /RCVR
WMQ Level 5.3.12 WMQ Level 5.1
OS Level 5.2 OS Level K1000
Windows / RCVR
WMQ Level 6.0.0.0
OS Level 2003 SP1
QMGR CCSID 1256
Problem: When a message is put with Arabic character Lam-

Alif it is not being converted properly between CCSID 420 and
CCISD 1089(Tandem) or 1256(Windows).

IBM Software Group

Solution :This is not a character conversion limitation for
WebSphere MQ, as not all characters can be converted
from one CCSID to another. In this case the Lam-Alif
character does not exist in CCSID/Code Page 1089 nor
1256, so a substitution character would be used, for
example X1A.
Enduser would need to specify a user defined message

format and write their own conversion code.
Reference: WebSphere MQ Application Programming

Reference (SC34-6062)

IBM Software Group
Problem Determination
Complete problem description
WMQ Environment description
WMQ Level, for example WMQ 6.0.2.2
OS level, for example iSeries 5.3.0; Windows XP SP2
QMGR CCSID, for example 37, 437, 5026, 1047
QMGR Sender Channel CONVERT setting
Putting Application MQMD CCSID and Format setting
Getting Application MQMD CCSID setting
Capture the message on the XMITQ.
Verify if it is correct.
Capture the message on the Destination Queue.
Verify if it is correct
WMQ Trace on both sending and receiving servers
IBM MustGather Technote : 1176955

IBM Software Group
IBM Technotes
MustGather: Documentation required by 1176955
WebSphere MQ data conversion
Data Conversion under WebSphere MQ 7005729
Conversion character differences between 1207860
CCSID 037 and CCSID 500
Data conversion in MQ intermittently does 1240018
not behave as expected
EBCDIC NL character is not converting 1116311
correctly in ASCII 0x15 (EBCDIC NL)
converts to 0xDA
AMQ6174 MQRC 2119 putting a message 1109867
to a remote queue

IBM Software Group
IBM Technotes cont
2110 MQRC Format Error 1232346
2080 MQRC TRUNCATED MSG FAILED 1167205
Data conversion does not occur in a 1268752

clustering environment
What CCSID is used by default for 1215215
WebSphere MQ client messages
MQ data conversion and data conversion 1167007
exit programs

IBM Software Group
Useful Links
WebSphere MQ Support
www.ibm.com/software/integration/wmq/support
System i Globalization
http://www.ibm.com/servers/eserver/iseries/software/globali
zation/
Unicode Home Page
http://www.unicode.org/
Character Sets and Code Pages, Encoding
http://www.i18nguy.com/unicode/codepages.html
Language Tools
http://www.google.com/language_tools?hl=en
http://babelfish.altavista.com/

IBM Software Group
Useful Tools
WebSphere MQ Supportpac
IH03 (IH03: WebSphere Message Broker V6-Message display,
test & performance utilities)
IBM Technote 4000637
RFHUTIL and RFHUTILC allows enduser to place test message
to a queue with a particular CCSID
WebSphere MQ Sample
AMQSBCG (distributed (Windows and Unix))
AMQSBCG4 (iSeries)
These programs are passed the name of a queue and a queue
manager. It then reads each message from the queue and
outputs the following to the stdout
Formatted message descriptor fields
Message data (dumped in hex and, where possible,
character format)

IBM Software Group
AMQSBCG/AMQSBCG4
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 273 CodedCharSetId : 819
Format : ' '
Priority : 0 Persistence : 0
MsgId : X'C1D4D840C3D6D4D4D6D540404040404046D2D73320001402'
CorrelId : X'000000000000000000000000000000000000000000000000'
BackoutCount : 0
ReplyToQ :' '
ReplyToQMgr : 'COMMON '
** Identity Context
UserIdentifier : 'STANLEY '
AccountingToken :
X'0F40404040404040404040404040404000000000000000000000000000
000008'
ApplIdentityData : ' '

IBM Software Group
AMQSBCG/AMQSBCG4 cont
** Origin Context
PutApplType : '8'
PutApplName : '021195/STANLEY/QP0ZSPWT '
PutDate : '20070903' PutTime : '18330499'
ApplOriginData : ' '
GroupId : X'000000000000000000000000000000000000000000000000'
MsgSeqNumber : '1'
Offset : '0'
MsgFlags : '0'
OriginalLength : '-1'
**** Message **** length - 52 bytes
00000000: 4A61 7661 2076 6572 7369 6F6E 206F 6620 '//.?>.?.'
00000010: 7468 6520 4D51 5055 5431 2061 7069 2020 '.(&../..'
00000020: 203A 20A3 A331 302E 3030 2024 2420 A2A2 '...tt.........ss'
00000030: 2023 2320 '.... '

IBM Software Group
z/OS Browse Sample

WebSphere MQ Support pac
MA17 (WebSphere MQ for z/OS Message
Handler Sample)
IBM Technote 4000071

IBM Software Group
z/OS MA17 browse sample

Queue Manager : RTP4 :
Queue : RTP4.QL :
Forward to Q Mgr : RTP6 :
Forward to Queue : :
Action : : (D)elete (F)orward
Message Content :
------------------------------------------------------------------------------
Message Descriptor
StrucId : `MD `
Version : 000000001
Report : 000000000
MsgType : 000000008
Expiry : -00000001
Feedback : 000000000
Encoding : 000000785
CodedCharSetId : 000000037
Format : `MQSTR `
Priority : 000000000
Persistence : 000000000
MsgId : `D4D8E3D7E4E361F2F0F0F7F0F8F3F0F1F6F4F9F3F6F9F1F6`X
CorrelId : `D4D8E3D7E4E361F2F0F0F7F0F8F3F0F1F6F4F9F3F6F9F1F6`X
BackoutCount : 000000002
ReplyToQ :` `
ReplyToQMgr : `RTP6 `

IBM Software Group
Documentation
The following manuals discuss MQ conversion topics. Just refer to
the index of these manuals for conversion or CCSID
WMQ Commands Manual Reference
WMQ Application Programming Reference
WMQ Application Programming Guide
Sample programs documented in this manual
WMQ Intercommunication
WMQ Conversion document
All manuals and documents can be found on the WebSphere MQ

Support webpage:

IBM Software Group
CCSIDs specified in this presentation

CCSID Description
37 USA, Canada, Netherlands, Portugal, Brazil, Australia, New
Zealand
420 Arabic EBCDIC
437 USA PC-Data
500 International EBCDIC
819 ISO 8859-1 ASCII
1047 Latin 1 (Open Systems)
1089 Arabic ISO 8859-6
1252 MS-WIN Latin-1
1256 MS-WIN Arabic

IBM Software Group
Summary
Conversion vs. Translation
CCSID (Coded Character Set Identifier)
Encoding
IBM Technotes
Useful Links
WebSphere MQ Product Support page
Search on Data Conversion to review IBM
Technotes regarding this topic

IBM Software Group
Additional WebSphere Product Resources

Discover the latest trends in WebSphere Technology and implementation,
participate in technically-focused briefings, webcasts and podcasts at:
www.ibm.com/developerworks/websphere/community/
Learn about other upcoming webcasts, conferences and events:
www.ibm.com/software/websphere/events_1.html
Join the Global WebSphere User Group Community: www.websphere.org
Access key product show-me demos and tutorials by visiting IBM Education
Assistant: ibm.com/software/info/education/assistant
Learn about the Electronic Service Request (ESR) tool for submitting
problems electronically:
www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf
.html
Sign up to receive weekly technical My support emails:
www.ibm.com/software/support/einfo.html

IBM Software Group
Questions and Answers

Websphere MQ Data Conversion PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Websphere MQ Data Conversion PDF

Uploaded by

Copyright:

Available Formats

IBM Software Group

WebSphere MQ Data Conversion

Blanche Ishida and Karen Stanley

WebSphere Support Technical Exchange

WebSphere MQ Message Header (MQMD)

Questions and Answers

WebSphere Support Technical Exchange 2

Terms and Definitions

WebSphere Support Technical Exchange 3

Terms and Definitions

CCSID Coded Character Set ID is a unique

WebSphere Support Technical Exchange 4

Terms and Definitions

WebSphere Support Technical Exchange 5

Note: MsgId, CorrelId, and Accounting Token are NOT

WebSphere Support Technical Exchange 6

WebSphere MQ Message Header

WebSphere Support Technical Exchange 7

WebSphere MQ Message Header cont

WebSphere Support Technical Exchange 8

D4C44040 00000001 00000000 00000008 MD ............

StrucId : 'MD ' Version : 2

WebSphere Support Technical Exchange 9

WebSphere Support Technical Exchange 10

Encoding scheme by platform

WebSphere Support Technical Exchange 11

To display the queue manager CCSID use the

WebSphere Support Technical Exchange 12

CCSID 437 -- this is an ASCII code page, used mainly under

WebSphere Support Technical Exchange 13

WebSphere Support Technical Exchange 14

WebSphere Support Technical Exchange 15

Problem : Sender channel fails with message AMQ9541

WebSphere Support Technical Exchange 16

Customer Scenario 1 cont

WebSphere Support Technical Exchange 17

Problem : Enduser indicated that they have an iSeries

WebSphere Support Technical Exchange 18

Customer Scenario 2 cont

Solution : Found that the receiver/responder

WebSphere Support Technical Exchange 19

Customer Scenario 2 cont

WebSphere Support Technical Exchange 20

What Gets Converted,?

Reference IBM Technote 7005729 : Data Conversion under

WebSphere Support Technical Exchange 21

Problem : Characters |, ] and ! are not being properly

WebSphere Support Technical Exchange 22

Customer Scenario 3 cont

Reference IBM Technote 1207860 : Conversion

WebSphere Support Technical Exchange 23

Code page 37 - Code page 500 differences

WebSphere Support Technical Exchange 24

Common problem code points

WebSphere Support Technical Exchange 25

WebSphere Support Technical Exchange 26

Customer Scenario 4 cont

Reference: WebSphere MQ Application Programming

WebSphere Support Technical Exchange 27

The message is entirely character data

Reference IBM Technote 7005729 : Data Conversion

WebSphere Support Technical Exchange 28

Common Procedure to ensure conversion on a