Infosphere CDC: System I

InfoSphere CDC
System i
© 2010 IBM Corporation

Information Management Software
CDC Architecture
2
CDC Log Based Architecture
Source Java-based GUI Target

Unified Admin Point
With Monitoring
ODS
Audit
Database
BI Appliance
TCP/IP
Message
Queue
Journal Log Publisher Engine Subscriber Engine Info

Redo/Archive Logs And Metadata And Metadata Server
Scrape Push Apply Confirm
3
CDC for System i Requirements
• Hardware Requirements
• Disk Space: 60 megabytes
• RAM: 64 megabytes of available pooled memory (512
megabytes recommended)
• Software Requirements
• Operating System: OS/400 V6R1 or greater
• For correct product operation, you may require Program
Temporary Fixes (PTFs). Contact IBM for PTF information for
your operating system.
4
CDC for System i Requirements (cont’d)
• Journal Requirements
• Journalling of *BOTH images
• Sufficient privileges to read journal
5
Authority Requirements
• When implementing the authority requirements for configuring and
operating CDC, there are 2 types of user profiles to take into consideration:
• CDC operational user:
• Users who run CDC replication processes
• Typically users who are logged on through the Management Console or who start replication
processes from the command line/job scheduler.
• The CDC source jobs are run under the CDC operational user profile.
• Requires authorities to the replicated source tables, journals and journal receivers.
• CDC owner/product user:
• By default, the user profile D_MIRROR that is automatically created during installation.
• CDC requires this user profile to supervise replication operations for the CDC apply
processes.
• The CDC apply processes are run under the D_MIRROR user profile, hence the need for
this profile to have update rights to the target tables.
• The password for this user profile is intentionally set to *NONE. This prevents users from
signing on interactively and also prevents the password from expiring which will stop CDC
apply processes.
6
CDC product library (*LIB) authorities

User Authority Level Comments
Product user *ALL Typically, D_MIRROR is the owner of the *LIB

(D_MIRROR or object
specified user
profile)
Operational user *USE Needed to create and access objects in the product
library
• Objects (*MSGQ, *USRQ) are created for new subscriptions. We

recommend that you set the CRTAUT parameter to *ALL to ensure that
all users entitled to use CDC can view event log messages and monitor
subscriptions activity
7
Replicated source table authorities
User Authority Comments

Level
Product user None The CDC product user does not need authorities
(D_MIRROR or to the replicated source tables (*FILE PF-
specified user DTA). A potential exception is when you wish
profile) to control replicated tables through the
D_MIRROR user profile (group profile,
adopted authority).
Operational user *USE, An operational user needs *USE authorities to
*OBJMGT refresh a table and to use it for continuous or
net change mirroring.
*OBJMGT authorities are required if journaling
for the tables must be started as part of
mapping the tables. If journaling is already
started, the *OBJMGT authority is not
required.
8
Journal authorities
(D_MIRROR or to the journal. A potential exception is when you
specified user wish to control replication through the
profile) D_MIRROR user profile (group profile, adopted
authority)
Operational user *OBJOPR, These authorities are required to start journaling
*OBJMGT, as part of mapping the tables, read the journal
*OBJEXIST, entries and the start the CDC subscription
*READ,
*ADD,
*UPDATE,
*EXECUTE
9
Journal receiver authorities

(D_MIRROR or to the journal receiver. A potential exception is
specified user when you wish to control replication through
profile) the D_MIRROR user profile (group profile,
adopted authority)
Operational user *OBJOPR, At different occasions (refresh table, start and end
*OBJMGT, of mirroring), user-defined journal entries are
*OBJEXIST, written into the journal to mark starting and
*READ, ending points. Add and update authorities are
*ADD, needed to be able to write these entries.
*UPDATE,
*EXECUTE
10
Target table authorities

Product user *CHANGE, Apply processes are run under this user profile,
(D_MIRROR or *OBJMGT hence the need for this profile to have update
specified user rights to the target tables. Object management
profile) authority is required to clear the target table
using CLRPFM in the event of a refresh
Operational user None By default, all apply processes are run under the
D_MIRROR user profile; no rights are needed for
the CDC operational user.
By default, all apply processes are run under the D_MIRROR user profile.
11
Expected jobs – initial

Connection to the
Agent from the
MC client
Listener – started
with the
subsystem
12
Expected jobs - Describe

Control job(s) ensure that:
• all jobs are properly
cleaned at shut down
• the communications link
is still up (Heartbeat)
DM*DESCRIB
programs are used
as part of the
describe process.
13

The communications
monitor job DMM#####MO
runs program DMCTSMO
overseeing all comms links
MONITOR job runs the

program DMMA which
services monitor
requests
14
There will be two jobs on each Source and

Target for the control link for comms.
DMS#####CI and DMS#####CO will appear for
each logical pair of source and target machine
and TCP ports
15
Expected jobs - Refresh

There will be two jobs on
each publisher and
subscriber, DMS#####DI and
DMS#####DO. The jobs are
used for the data link for each
subscription that is running
16
Expected jobs – Refresh (2)

The DMTAPPLY job is
responsible for applying
data on the target
The DMSREFRESH
program is used as
part of the refresh
process.
17
Expected jobs - Mirror

So far, the jobs are
identical to what was
seen during a
Refresh…
18
Expected jobs – Mirror (2)

DMSMIRROR
program is used as
part of the mirroring
process.
The DMSSCRAPER is
responsible for
scraping the
replication log
(journals) for changes.
19
Expected jobs – Mirror (3)
20
The replication log
• OS/400 journals are used for the CDC replication log on the
System i platform.
• Efficient way to capture changes to DB2/System i databases.
• The journal is a secure database log, it cannot be modified.
• All database events are logged in a sequential manner with a
unique sequence number
• Each entry has a unique code and type
• Typical Codes: R = Record, F = File
• Typical Types: PT = Put/Add, DL = Delete
• Up to 10 journals per subscription are supported.
21
How does journaling work?

• 1 file : 1 journal
• 1 journal : Many files
Journal (*JRN)
e.g.. DMCJRN
Physical File (*FILE, PF)
On-line
On-line
1. R PT Add Attached = Current
2. R DL Delete
3. R PT Add On-line = Attached, not current
4. R PT Add Saved = As above, but saved
5. ….. Damaged = Error on chain
Journal Receiver (*JRNRCV) – ASP, e.g. DMCJRN0001, DMCJRN0002...
22
Journal management - Default journal
Change journal receiver should be run regularly

• CHGJRNDM Command (at prompt or in a Scheduler)
The CHGJRNDM command performs the following actions:

• Changes receiver
• Removes the old receiver
23
User defined journal set-up

To use a user-defined journal, set up the journaling for the
file prior to selecting the file for replication.
Required:
• Record both the before and after images for updates
(*BTH).
• Set Delete = *NO for the journal receivers
Recommended:
• Omit journal entries for opens and closes (OMTJRNE
*OPNCLO).
• Add a threshold.
• Allow *SYSTEM to manage the receivers.
24
Journal management - user defined
When working with user-defined journals it is recommended

that you:
• Change journal receivers regularly
• Monitor and delete old receivers
It is also recommended that a CL is created which:

• Runs the RTVDMJENT command
• Runs a CHGJRN command
• Delete old receivers (DLTJRNRCV)
25
Journal management - user defined

What if you want to have
this utility clean up a
different journal?
• View the
DMIRROR/DMFS503
P
• All journals with a
clean-up method of
“1” will use the utility.
• Alter any journal you
wish to clean up
currently with a value
of “3” to a value of “1”.
26
Monitoring
Spool Files
• Monitor spool files produced under the submitter account

and the DMIRROR account.
• A periodic purge of spool files as part of your standard

system clean-up process is recommended.
• To reduce the amount of messages being logged, set the

message logging level to *NOLIST (not recommended).
• The spool files will only catch job logs of priority>20 or jobs
ending abnormally.
• CHGJOBD JOBD(DMCJOBD) LOG (4 00 *NOLIST)
27
Monitoring
Message Queues
• CDC produces messages on both the Source and the

Target.
• A periodic clean-up of the message queues is

recommended.
28
Monitoring current bookmark position
• DMVWBKMK
• With the bookmark changes made in CDC users have lost the
ability to access the bookmark(s) via the DMFT504P.
• DMVWBKMK retrieves bookmark information from the
JRN_STATUS userspace in the CDC product library.
• This command can be invoked on a Target.
29
Monitoring
Messages
The DMUMESSAGE file, located in the installation directory,

contains all messages used by CDC.
E.g. DMU0601 = “TCP/IP error occurred”
30
Monitoring
• NOTIFICATION (Alerts & Alarms)
31
Monitoring
• Progress Messages
32
Operational considerations
© 2010 IBM Corporation

Commands
• For an explanation of available commands, refer to

Chapter 7 (Operational and Maintenance Issues) in the
CDC for System i User Manual.
• When executing any of the commands, set the current

library (CHGCURLIB) to the CDC installation library.
• Sample CLs are available in the CDC System i User

Manual.
34
Determining the current version and PackID
• To determine your current version and patch (packid), use

the following commands:
• DSPDTAARA D_VERSION
• DSPDTAARA DMPACKID
35
Back-ups
• DMIRROR, which contains the installation library and the
metadata, should be periodically backed-up.
• It is critical to perform back-ups after any additions or changes
have been made.
36
Tuning
System level:
• Change the CDC subsystem to utilize shared pools.
• Table level:
• Use multiple journals - by journal splitting, there are more
processes and throughput is increased.
• Look at what and how you are journaling. Are you omitting
opens and closes?
• Create multiple subscriptions, which will create multiple
processes.
37
Automation on System i
• When automating on System i, one can consider:
• Available commands
• Common and Useful jobs

• Autostart
• Shutdown
• Journal receiver clean-up
38
CDC for System i commands
• A number of commands are available from CDC for System

i
• Designed to make set-up and administration easier and
faster.
• Provides operational control of replication.
• All commands can be invoked from the command entry
line, included in a job scheduler or a CL program.
• Replication can be scheduled to run automatically once
initial definitions are completed.
• NOTE: It is important to make the CDC product library the
current library when running commands/CLs.
39
CDC for System i commands

Commands Functions
STRDTAMIR Describe Source Tables to Target System
STRDTAMIR Start Continuous Mirroring
ENDDTAMIR End Continuous Mirroring
STRDTAMIR Start Net Change Mirroring
ENDDTAMIR End Net Change Mirroring
STRDTARSH Start Data Refresh
ENDDTARSH End Data Refresh
40
Operational commands
Commands Functions
RTVDMJENT Retrieve Transformation Server Journal Entry
SETTSSYNC Set Synchronization Point
VFYTSJRN Verify Journal
SETJRNPOS Set Journal Position
CHGJRNDM Change DataMirror Journal Receiver
DMVWBKMK View Bookmark Position
RTVDMVSTS Retrieve Transformation Server Version Status

CHGREPMTD Change Replication Method
SETRSHSTS Set Refresh Table Status
SETMIRSTS Set Mirrored Table Status
41
Operational commands
Commands Functions
STRDMTCP Start TS TCP/IP Communications
ENDDMTCP End TS TCP/IP Communications
STRDMUDP Start DataMirror UDP Listener
ENDDMUDP End DataMirror UDP Listener
DMVFYTCP COMM Verifier for TCP/IP
42
Setup and administration commands

Commands Functions
ADDTBL Add Tables
RMVTBL Remove Tables
SLTTGTTBL Select Target Tables
DSLTGTTBL De-Select Target Tables
ASNTGTTBL Assign Source Tables To Target
DASTGTTBL De-Assign Target Tables
CPYTGTTBL Copy Target Tables
CPYTGTASN Copy Target Assignments
DUPTBL Duplicate Tables
DUPTGTTBL Duplicate Target Tables
DUPTGTASN Duplicate Target Configuration
43
Example of how to call a CDC command:
• ENDDTAMIR TARGET() [ENDTYP()]

• ENDDTARSH TARGET() [ENDTYP()]
• TARGET Target system name
• Name of subscription
• Generic name of subscription, ie. SUB* specifies subscriptions starting
with ‘SUB’
• *ALL
• ENDTYP Indicates how you want to end replication
• *CNTRLD Refresh/Mirroring is ended in a controlled manner
• *IMMED Refresh/Mirroring is ended Immediately
• E.g.
• ENDDTARSH TARGET(*ALL) ENDTYP(*CNTRLD)
• ENDDTAMIR TARGET(‘TARGET*’) ENDTYP(*IMMED)
44
Shutting down CDC

• It is preferable to stop CDC before a backup and/or IPL
• To do this, add the following lines to the existing shut down

routine:
0002.00 CHGCURLIB CURLIB(<CDC install library>)

0003.00 ENDDTAMIR TARGET(*ALL)
0004.00 MONMSG(CPF0000)
• You can use these in any scheduler - IBM's or any third

party tool
45
Re-starting CDC after a shutdown

• To start replication after an IPL or backup, add the following
two lines:
0002.00 CHGCURLIB CURLIB(<CDC install library>)
0003.00 STRDTAMIR TARGET(*ALL) IDLE(*WAIT)
0004.00 MONMSG(CPF0000)
• Start individual subscriptions by using:

0003.00 STRDTAMIR TARGET(<target name>) IDLE(*WAIT)
0004.00 MONMSG(CPF0000)
• You can use these in any scheduler - IBM's or any third party
tool
46
Adding in Journal Clean-up Routines
• Journal clean-up is a critical maintenance task
• How often is journal clean-up configured to run in the

environment?
• Upon automated start-up, some clients take the opportunity

to perform journal receiver clean-up
47
Default Journal Clean-up Routines

• Are you using the default journal, DMCJRN?
• CDC provides a command for default journal clean-up
• To clean journal receivers which are no longer required,

use the following command:
0003.00 CHGJRNDM
48
CHGJRNDM – Change Journal Receivers

• Allows the user to generate new journal receivers for a journal on
the source server or delete processed and obsolete journal
receivers
• Command Syntax
CHGJRNDM JOURNAL() [CHGRCV()] [DLTRCV()]
• Parameters
• JOURNAL The name of the journal for which journal
receivers are being generated. (*ALL to
specify all journals used by CDC)
• CHGRCV Indicate whether the receivers should be
changed (*YES, *NO)
• DLTRCV Indicate whether the obsolete receivers should be
deleted for the specified journal
49
DMVWBKMK – Retrieve Bookmark Information

• Use this command to retrieve the CDC bookmark information from the
JRN_STATUS user space in the product library
• The bookmark will indicate the progress of CDC scraping and applying
activities for a subscription and retrieve exact information about the journal
or log position that is being processed on the target
• Command Syntax:
• DMVWBKMK [PRODLIB()] PUBID() [OUTPUT()] [OUTFILE()] [MBROPT()]
• Applies to Target
• Parameters
• PRODLIB The name of the CDC product library
• PUBID The name of the source identifier (subscription)
• OUTPUT Indicates where the output is directed (*DSP, *OUTFILE)
• MBROPT Indicates whether the database file specified for the OUTFILE is
cleared before new records are added to the file (*REPLACE, *ADD)
50
Result of DMVWBKMK
Last Committed Position
The position of the last journal or log entry to be applied to the target table that is
confirmed. Confirmed indicates either that the commitment control group where
the journal or log entry belongs has been committed, or that the journal or log
entry was processed outside commitment control.
Earliest Open Position
The position marking the beginning of the oldest commitment control transaction
that is still uncommitted or being rolled back.
Last Applied Position
The position of the last journal or log entry applied to the target table.
Apply Time
The timestamp of the last journal or log entry applied to the target table (see Last
Applied Position above).
Log Time
The timestamp in the publication journal or log of the last journal or log entry
applied to the target table
51
SETJRNPOS – Set Journal Position

• Use this command to position CDC to start mirroring at a specific journal
entry
• Command Syntax:
• SETJRNPOS JOURNAL() TARGET() JRNRCVNMA() {STRDTE() STRTIM() |
STRSEQNBR()}
• Applies to Source
• Examples:
• SETJRNPOS JOURNAL(D_MIRROR/DMCJRN) TARGET(SUB1)
JRNRCVNME(D_MIRROR/DMCRCV004)STRDTE('07052004')
STRTIM('141200')
The search for the specified entry starts with journal receiver
D_MIRROR/DMCRCV004 associated with journal D_MIRROR/DMCJRN.
Positions CDC to start mirroring at the entry in a journal receiver that was
timestamped on July 5, 2004 at 2:12 PM.
• SETJRNPOS JOURNAL(D_MIRROR/DMCJRN) TARGET(SUB2)
JRNRCVNME(D_MIRROR/DMCRCV006)STRSEQNBR(127855)
The search for the specified entry starts with journal receiver
D_MIRROR/DMCRCV006 associated with journal D_MIRROR/DMCJRN.
Positions CDC to start mirroring at the entry that has a sequence number of
127855.
52
QUESTIONS?
53

Infosphere CDC: System I

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Infosphere CDC: System I

Uploaded by

Copyright:

Available Formats

InfoSphere CDC

© 2010 IBM Corporation

CDC Log Based Architecture

Source Java-based GUI Target

Journal Log Publisher Engine Subscriber Engine Info

Scrape Push Apply Confirm

CDC for System i Requirements

CDC for System i Requirements (cont’d)

CDC product library (*LIB) authorities

Product user *ALL Typically, D_MIRROR is the owner of the *LIB

• Objects (*MSGQ, *USRQ) are created for new subscriptions. We

Replicated source table authorities

User Authority Comments

Journal receiver authorities

Target table authorities

Expected jobs – initial

Expected jobs - Describe

Expected jobs - Describe

MONITOR job runs the

Expected jobs - Describe

There will be two jobs on each Source and

Expected jobs - Refresh

Expected jobs – Refresh (2)

Expected jobs - Mirror

Expected jobs – Mirror (2)

Expected jobs – Mirror (3)

The replication log

How does journaling work?

Journal Receiver (*JRNRCV) – ASP, e.g. DMCJRN0001, DMCJRN0002...

Journal management - Default journal

Change journal receiver should be run regularly

The CHGJRNDM command performs the following actions:

User defined journal set-up

Journal management - user defined

When working with user-defined journals it is recommended

It is also recommended that a CL is created which:

Journal management - user defined

• Monitor spool files produced under the submitter account

• A periodic purge of spool files as part of your standard

• To reduce the amount of messages being logged, set the

• CHGJOBD JOBD(DMCJOBD) LOG (4 00 *NOLIST)

• CDC produces messages on both the Source and the

• A periodic clean-up of the message queues is

Monitoring current bookmark position

The DMUMESSAGE file, located in the installation directory,

E.g. DMU0601 = “TCP/IP error occurred”

© 2010 IBM Corporation

• For an explanation of available commands, refer to

• When executing any of the commands, set the current

• Sample CLs are available in the CDC System i User

Determining the current version and PackID

• To determine your current version and patch (packid), use

• Common and Useful jobs

CDC for System i commands

• A number of commands are available from CDC for System

CDC for System i commands

RTVDMVSTS Retrieve Transformation Server Version Status

Setup and administration commands

Example of how to call a CDC command:

• ENDDTAMIR TARGET() [ENDTYP()]

Shutting down CDC

• To do this, add the following lines to the existing shut down

0002.00 CHGCURLIB CURLIB(<CDC install library>)

Product user ALL Typically, D_MIRROR is the owner of the LIB

• Objects (MSGQ, USRQ) are created for new subscriptions. We