IBM Software Group

WebSphere MQ Data Conversion

Part II

Blanche Ishida and Karen Stanley

WebSphere MQ Support

WebSphere Support Technical Exchange

 IBM Software Group

Agenda

Terms and Definitions

WebSphere MQ Message Header (MQMD)

Customer Scenarios

IBM Technotes

Useful Links

Summary

Questions and Answers

WebSphere Support Technical Exchange 2

 IBM Software Group

Recap: Terms and Definitions

Character Set

Is a defined set of characters used to represent textual
information (e.g. 0-9, a-z, A-Z, .,;:!?/-)

A Character Set generally supports more than one
language

Code Page

(AKA Code set) A code page can be defined as a table
with a mapping of alphanumeric code and its binary
representation.

Where each character in a character set is assigned a
numerical representation (often used interchangeably
with character set e.g. charset in HTML)

WebSphere Support Technical Exchange 3

 IBM Software Group

Recap: Terms and Definitions

A code point is the location of a character within
the code page

CCSID Coded Character Set ID is a unique

number (0-65535) used by IBM to uniquely identify
a Coded Character Set and a Codepage.

WebSphere Support Technical Exchange 4

 IBM Software Group

Recap: Terms and Definitions

ASCII : American Standard Code for Information Interchange.
The code used as the basis for most codes apart from
EBCDIC.

EBCDIC : Extended Binary Coded Decimal Information Code
(EBCDIC). It is typically used on z/OS, iSeries and VSE

UNICODE : A code page which defines most of the
characters used by the languages now used throughout the
world. UCS-2, UTF-8, UTF-16 and UTF-32 for encodings.

SBCS Single Byte Character Set. A code page with up to 256
characters which can be represented by a single 8 bit byte.

DBCS Double Byte Character Set. A code page with up to
65536 characters.

WebSphere Support Technical Exchange 5

 IBM Software Group

Recap: MQMD

The Message Descriptor is not entirely character data, so
some of it will be converted as character data, some as
numeric, and some is passed unconverted.

Note: MsgId, CorrelId, and Accounting Token are NOT

considered to be character data, and as such WILL NOT be
converted as character data. Many applications put
character data into these fields, but if the message is being
exchanged between ASCII and EBCDIC platforms, and the
application on the other side needs to be able to read these
fields, then it is the responsibility of the application (or a user
exit) to convert these fields. Also the data maybe in
hexadecimal representation for MsgID and CorrelID.

WebSphere Support Technical Exchange 6

 IBM Software Group

Recap: WebSphere MQ Message Header

Encoding Numeric encoding of message data

MQMD offset X18

CodedCharSetId Character set identifier of message data

MQMD offset X1C

Format Format name of message data

MQMD offset X20

MsgId Message identifier

MQMD offset X30

CorrelId Correlation identifier

MQMD offset X48

WebSphere Support Technical Exchange 7

 IBM Software Group

Recap: MQMD : Encoding

Encoding is generally taken to mean the method
that this platform uses to represent numeric data.
There are two general types:

LittleEndian, used by Intel processors (e.g
Windows, Linux on Intel). e.g. the number 437
would be represented (in hex) as XBF01 (we
refer to this as byte swap)

BigEndian. e.g. the number 437 would be
represented (in hex) as X01BF.

WebSphere Support Technical Exchange 8

 IBM Software Group

Recap: CCSID

The Queue Manager CCSID is important as it
defines the default CCSID for messages being
created using MQPUT and is the default CCSID
which will be used for conversion when messages
are retrieved using MQGET. If the channel option
CONVERT(YES) is set the conversion is attempted
for the receiving Queue Managers CCSID.

To display the queue manager CCSID use the

DIS QMGR command

WebSphere Support Technical Exchange 9

 IBM Software Group

Recap: MQMD : CodedCharSetId

Some well known CCSIDs:

CCSID 437 -- this is an ASCII code page, used mainly under

OS/2, DOS, and Microsoft Windows console (OEM) windows.

CCSID 850 -- this is another common ASCII code page, used
mainly on under OS/2, DOS, and Microsoft Windows console
(OEM) windows used on PCs in Europe and some AIX
locales.

CCSID 819 -- this is the ISO 8859-1 standard Western
European ASCII code page. Used by most UNIX locales.

CCSID 500 -- This is an EBCDIC code page, used mainly on
z/OS. It is known as the International codepage.

CCSID 037 -- This is another popular EBCDIC code page,
used on OS/400. It is the US English codepage.

WebSphere Support Technical Exchange 10

 IBM Software Group

Recap: FORMAT

The queue manager built-in formats all have
names beginning with MQFMT. They are listed and
described in the WebSphere MQ Application
Programming Reference under the Format field of
the Message descriptor (MQMD)

Some built-in formats

MQFMT_NONE (default)
MQFMT_STRING (for character data conversion)

WebSphere Support Technical Exchange 11

 IBM Software Group

Recap: What Gets Converted?

For Channel Communication -- When a channel between two
WebSphere MQ Queue Managers is started, the two Queue Managers
will negotiate which CCSID and encoding protocol they will use for
channel communications.

Reference IBM Technote 7005729 : Data Conversion under

WebSphere MQ

WebSphere Support Technical Exchange 12

 IBM Software Group

Recap: Customer Scenario 1

Windows/Sender
z/OS /Receiver

WMQ Level 6.0.?.?
WMQ Level 6.0

OS Level ???
OS Level ???

QMGR CCSID 437
QMGR CCSID 500

Problem : Characters |, ] and ! are not being properly

handled when sent from Windows to z/OS.

Reference trace file : CCSID37.CCSID500.doc

WebSphere Support Technical Exchange 13

 IBM Software Group

Recap: Customer Scenario 1 cont

Solution : Change the queue manager on z/OS to
CCSID 37 or a create a channel exit program that will
handle the conversion.

Reference IBM Technote 1207860 : Conversion

character difference between CCSID 37 and 500

WebSphere Support Technical Exchange 14

 IBM Software Group

Code page 37 - Code page 500 differences

WebSphere Support Technical Exchange 15

 IBM Software Group

Common problem code points

Character CCSID 437 CCSID 819 CCSID 500 CCSID 37
! 21 21 4F 5A
AA AC BA 5F
9B A2 B0 4A
| 7C 7C BB 4F
[ 5B 5B 4A BA
] 5D 5D 5A BB
^ 5E 5E 5F B0

WebSphere Support Technical Exchange 16

 IBM Software Group

Customer Scenario 2

Windows WMQ Client
z/OS /SVRCONN

WMQ Level 6.0.x
WMQ Level 6.0

OS Level ???
OS Level ???

Putting Application CCSID 437
QMGR CCSID 500

Channel defined with CONVERT=Y
Must have Client attachment

Problem: Messages put to z/OS queue with a CCSID of 437 are not
converted as expected.

WebSphere Support Technical Exchange 17

 IBM Software Group

Customer Scenario 2 cont

Solution : Messages PUT to the z/OS queue manager
or any platform should have MQFMT_STRING format
specified in the message header (MQMD).

Reference: WebSphere MQ Application Programming

Reference (SC34-6062)

Reference trace file :

CCSID437.CCSID500.NO.MQSTR.doc

WebSphere Support Technical Exchange 18

 IBM Software Group

The message is entirely character data

If the message you are passing is entirely character data then it is
very easy to have WebSphere MQ do all of the work for you. In the
application that is doing the MQPUT of the message, you should
set MQMD.Format to MQFMT_STRING. MQFMT_STRING is a
constant that equates to the string value MQSTR . This value in
the Format field will let WebSphere MQ know that the message
consists entirely of character string data.

Reference IBM Technote 7005729 : Data Conversion

under WebSphere MQ

WebSphere Support Technical Exchange 19

 IBM Software Group

Recap: Common Procedure to ensure conversion

on a MQGET

In the application doing the MQGET of the message, set MQMD.CodedCharSetId to
0 to pick up the default CCSID of the QMGR. This will ensure that WebSphere MQ
knows the correct CCSID to convert to.

In the application that is doing the MQGET of the message, make sure you specify
MQGMO_CONVERT as one of the MQGMO.Options. This field has many options
that should be either added or bitwise ORed together if you need to specify more
than one.

It is recommended that you always do the data conversion on the MQGET of the
message, since the message may take many hops across different platforms on its
way to its destination, and doing data conversion on each hop is a waste of
resources.

If you specify CONVERT(YES) on the SENDER channel that sends the data to its
final destination, WebSphere MQ will convert the data into the CCSID of the queue
manager on the target system.

WebSphere Support Technical Exchange 20

 IBM Software Group

Customer Scenario 3

CICS TS 3.2
z/OS

Using CICS Bridge
WMQ Level 6.0

OS Level ???

QMGR CCSID 500

Problem: No data conversion, binary zeroes returned, bad

messages, Abends, unpredictable results.

Solution : Apply CICS TS apar PK58227 AND WMQ apar PK59397

Local Fix : As a temporary work around for this issue change the
Sender channel definition to have YES for the CONVERT parameter.

WebSphere Support Technical Exchange 21

 IBM Software Group

Customer Scenario 4

Linux/SDR
iSeries/ RCVR

WMQ Level 6.0.1.1
WMQ Level 6.?.?.?

OS Level ??
OS Level ???

QMGR CCSID 819
QMGR CCSID 932

Problem: When a message is sent from Linux (819) to iSeries (932)

not all data is converted as expected. A return code of 2119 is
generated MQRC_NOT_CONVERTED.

Solution : Data conversion from code page 819 to 932 is not

supported. Customer will need to use an alternate CCSID, for
example 1208 (unicode).

Reference trace file : MQRC_NOT_CONVERTED.doc

WebSphere Support Technical Exchange 22

 IBM Software Group

Customer Scenario 5

Windows Client
iSeries/ SVRCN

WMQ Level 6.0.2.4
WMQ Level 6.0.2.4

OS Level ??
OS Level 5.3

QMGR CCSID ???
QMGR CCSID 37

Application CCSID 819
Application CCSID 1208

Problem: Messages put to iSeries queue with a CCSID of 819 are not converted
as expected. Server application has get option of MQGMO_CONVERT specified for
the MQGET api. Application fails with a RC 2190
MQRC_CONVERTED_STRING_TOO_BIG. Messages are not retrieved from the
queue.

Solution : Conversion between 819 (SBCS) and 1208 (MBCS) will fail if
application buffer size is not large enough to handle data. Verify the size of the
receiving buffer length. Remember when converting from a SBCS to MBCS to
double the size of the buffer.

Reference trace file : MQRC_CONVERTED_STRING_TOO_BIG.doc

WebSphere Support Technical Exchange 23

 IBM Software Group

Customer Scenario 6

Windows Client

WMQ Level 6.0.2.4

OS Level ??

QMGR CCSID ???

Application CCSID 819

Data Length 25 Bytes

iSeries/ SVRCN

WMQ Level 6.0.2.4

OS Level 5.3

QMGR CCSID 37

Application CCSID 1208

Buffer Length 24 Bytes

MQGET options: Convert, Accept
Truncated message

Problem: Messages put to iSeries queue with a CCSID of 819 are not
converted as expected. Server application has get options of
MQGMO_CONVERT and MQGMO_ ACCEPT_TRUNCATED_MSG
specified for the MQGET api, which fails with a RC 2079
MQRC_TRUNCATED_MSG_ACCEPTED. However, the messages are
retrieved successfully, but not all data is converted.

WebSphere Support Technical Exchange 24

 IBM Software Group

Customer Scenario 6 cont.

Solution : Conversion between 819 (SBCS) and 1208
(MBCS) will fail if application buffer size is not large enough
to handle data. In this scenario the customer has specified
MQGET option MQGMO_ACCEPT_TRUNCATED_MSG ,
which will allow the message to be retrieved from the queue.
However, the buffer length specified for the application
cannot hold the entire converted data. Verify the size of the
receiving buffer length. Remember when converting from a
SBCS to MBCS/DBCS to double the size of the buffer.

Reference trace file :

MQRC_CONVERTED_MSG_TOO_BIG.doc

WebSphere Support Technical Exchange 25

 IBM Software Group

What Unicode CCSID can be used?

Reference IBM Technote 7005729: Data
Conversion under WebSphere MQ

1200

1208

13488

NOTE: There are many unicode CCSIDs, however

you must select which one is best for your
environment.

WebSphere Support Technical Exchange 26

 IBM Software Group

Problem Determination

Complete problem description

WMQ Environment description

WMQ Level, for example WMQ 6.0.2.2

OS level, for example iSeries 5.3.0; Windows XP SP2

QMGR CCSID, for example 37, 437, 5026, 1047

QMGR Sender Channel CONVERT setting

Putting Application MQMD CCSID and Format setting

Getting Application MQMD CCSID setting

Capture the message on the XMITQ.
Verify if it is correct.

Capture the message on the Destination Queue.
Verify if it is correct

WMQ Trace on both sending and receiving servers

IBM WebSphere MQ Data Conversion : Technote 7010477

WebSphere Support Technical Exchange 27

 IBM Software Group

Useful Links

WebSphere MQ Support

www.ibm.com/software/integration/wmq/support

System i Globalization

http://www.ibm.com/servers/eserver/iseries/software/globali
zation/

Unicode Home Page

http://www.unicode.org/

Character Sets and Code Pages, Encoding

http://www.i18nguy.com/unicode/codepages.html

Language Tools

http://www.google.com/language_tools?hl=en

http://babelfish.altavista.com/

WebSphere Support Technical Exchange 28

 IBM Software Group

Useful Tools

WebSphere MQ Supportpac

IH03 (IH03: WebSphere Message Broker V6-Message display,
test & performance utilities)
IBM Technote 4000637
RFHUTIL and RFHUTILC allows enduser to place test message
to a queue with a particular CCSID

WebSphere MQ Sample

AMQSBCG (distributed (Windows and Unix))

AMQSBCG4 (iSeries)
These programs are passed the name of a queue and a queue
manager. It then reads each message from the queue and
outputs the following to the stdout
Sample programs do not have MQGMO_CONVERT
specified
Formatted message descriptor fields
Message data (dumped in hex and, where possible,
character format)

CSQ4BCL1 (z/OS)

WebSphere Support Technical Exchange 29

 IBM Software Group

IH03: Message from i5/OS queue manager

WebSphere Support Technical Exchange 30

 IBM Software Group

WebSphere Support Technical Exchange 31

 IBM Software Group

WebSphere Support Technical Exchange 32

 IBM Software Group

Message from z/OS queue manager

WebSphere Support Technical Exchange 33

 IBM Software Group

iconv API

On the i5/OS(qshell), AIX, Linux and z/OS (OMVS)
platforms, enduser can take advantage of the iconv
unix api/command to verify data conversion.

Example from the i5/OS platform:

$
> iconv
iconv: 001-0098 Required argument is not specified.
iconv: 001-3051 usage: iconv -f fromCCSID -t toCCSID [file ...]
$

NOTE: On the i5/OS platform, use the DSPF command with F10=Display Hex to
view data in hexdecimal format.

WebSphere Support Technical Exchange 34

 IBM Software Group

iconv API cont

> iconv -f 37 -t 37 ccsidfile
this is a test message. Now is the time for all good men to come to the aid of their country.
> iconv -f 37 -t 819 ccsidfile
$ / _/ +? _ ? /%% ?? _> ? ?_ ? / ?
?>`
> iconv -f 37 -t 437 ccsidfile
$ / _/ +? _ ? /%% ?? _> ? ?_ ? / ?
?>`
> iconv -f 37 -t 500 ccsidfile
this is a test message. Now is the time for all good men to come to the aid of their country.
> iconv -f 37 -t 1200 ccsidfile
$ / _/ +? _ ? /%% ?? _
> ? ?_ ? / ? ?>`
> iconv -f 37 -t 1208 ccsidfile
$ / _/ +? _ ? /%% ?? _> ? ?_ ? / ?
?>`

WebSphere Support Technical Exchange 35

 IBM Software Group

Documentation

The following manuals discuss MQ conversion topics. Just refer to
the index of these manuals for conversion or CCSID

WMQ Commands Manual Reference

WMQ Application Programming Reference

WMQ Application Programming Guide
Sample programs documented in this manual

WMQ Intercommunication

WMQ Conversion document

All manuals and documents can be found on the WebSphere MQ

Support webpage:
www.ibm.com/software/integration/wmq/support

WebSphere Support Technical Exchange 36

 IBM Software Group

Summary

Terms and Definitions

IBM Technotes

7010477 Webcast replay: WebSphere MQ Data
Conversion

Useful Links

WebSphere MQ Product Support page
www.ibm.com/software/integration/wmq/support
Search on Data Conversion to review IBM
Technotes regarding this topic

WebSphere Support Technical Exchange 37

 IBM Software Group

Additional WebSphere Product Resources

Discover the latest trends in WebSphere Technology and implementation,
participate in technically-focused briefings, webcasts and podcasts at:
www.ibm.com/developerworks/websphere/community/

Learn about other upcoming webcasts, conferences and events:
www.ibm.com/software/websphere/events_1.html

Join the Global WebSphere User Group Community: www.websphere.org

Access key product show-me demos and tutorials by visiting IBM Education
Assistant: ibm.com/software/info/education/assistant

Learn about the Electronic Service Request (ESR) tool for submitting
problems electronically:
www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf
.html

Sign up to receive weekly technical My support emails:
www.ibm.com/software/support/einfo.html

WebSphere Support Technical Exchange 38

 IBM Software Group

Questions and Answers

WebSphere Support Technical Exchange 39