UCM CDI Integrating Application

Tata Consultancy Services ltd.

December 1, 2010

COURSE AGENDA
Day 1 Module 1 : Review of CDI Architecture Module 2 : Overview of Application Integration Module 3 : Initial Data Loads into UCM Module 4 : The UCM Batch Workflow and SDH tables Day2 Module 5 : UCM Messaging and CRMML Module 6 : UCM Server Party Package Workflows Module 7 : The UCM Batch Management Workflows Module 8 : Publish and Subscribe Module 9: Wrap-up

Module 1: Review of Siebel CDI Architecture

The CDI Solution

The Customer Data integration(CDI) solution consists of three products

CDI

UCM

SDQ

UAN

UCM = Universal Customer Master

What is UCM? A Siebel horizontal application which is the central master database/application for storing any and all customer information . What it does? 1. Acts as a central repository of customer data. 2. Keeps customer information in synch with other systems in the enterprise, thus giving a 360 real-time true picture of the customer. In a typical large enterprise, you have hundreds of various proprietary and legacy applications which deal with different parts of the customer information , some deal with service data, some with sales data, some others purely with demographics and contact data. UCM has the capability to synchronize and bring together all this data to give a single unified view of the customer as a whole.

SDH = Source Data and History tables

The central components of the CDI solution are the Source Data and History (SDH) tables associated with UCM 1. Provide a mechanism to maintain audit-trace of changes happening to the customer record. 2. Provide a staging area for data quality. External systems first integrate with these SDH tables when sending data to UCM, UCM will then in turn use its data quality rules to sort out the good updates from the bad ones and apply the good ones to the base tables while storing the bad ones too for audit purposes. E.g. In an enterprise, suppose you have a .NET based sales application and a legacy service application, both storing the customers contact number. If the enterprise feels that its service department is more likely to have a more current contact number of the customer, you can define the service application with a higher confidence level for updating this field. In cases where both applications are sending simultaneous updates to this field, both updates will be stored in the SDH tables, but the one from the 6

UCM Data Model

Is an extension of Siebel Data model Source, Data and history tables are added. UL
List Column/ Control Applet View Screen Application

BL

.
Field BC BO

DL

SDH Column

SDH table

Source Data and History types

SDH data contains several types of records distinguished by a Type value Each record is one of the following types: 1. History 2. Source 3. Merged 4. Cleansed 5. Queued 6. Unmerge 7. Batch A single transaction typically results in the creation of multiple Source data and history records.

Source Data and History : Data schema

Provide the infrastructure necessary to support child structures Example : addresses for contacts
S_ORG_EXT Base table Source data History Table S_UCM_ORG_EXT S_UCM_CONTACT S_CONTACT

S_CIF_EXT_SYST

S_UCM_ORG_CHLD

External System information table

S_UCM_CON_CHLD

Source Data History Child table

9

UCM Application Integration

Revolves around integrating to these SDH tables First basic CDI configuration must be performed 1. Systems registered 2. Data quality configured 3. Survivorship configured

10

CDI Configuration
External System Registration Data Quality Configuration Survivorship Rules

11

External System Registration

Every operational application that connects to UCM must register with UCM. After registration, a systems privileges and accessibility can be defined 1. Query 2. Insert 3. Update 4. Delete Defines grant access to Siebel UCM Privileges are granted at the BO layer

12

How to register an external application with UCM

Use the system registration view 1. Select Administration Universal Customer Master > System registrations 2. Click New 3. Enter a system name, id and protocol type(HTTP,MQSeries) 4. Below System registrations applet Select New and assign BO privileges in theSystem Privileges.

13

Registering an External Application

14

External System referencing: CrossReferencing

Cross-referencing stores source identification information for data in the UCM UCM stores identification information about external systems that have been registered. Cross-referencing provides a one-to-many mapping of customer records across multiple systems Allows UCM to keep track of the same customer record across multiple systems A single UCM record maps to one or many records in any registered application UCM supports cross-referencing for the following objects: 1. Contacts 2. Accounts 3. Households
15

Cross-Referencing
Cross-reference entries are created automatically with the creation of every new account, contact, or household record External systems send requests to insert a record If there is an existing cross-reference entry, the entry is updated If there is no existing cross-reference entry, a new record is inserted The system ID is the value assigned when the system was registered and system name is the associated name There are multiple possible external Ids to account for situations where multiple records in an operational application map to a single record in UCM Only the first is stored in the ID tag in a message request, by default

16

External System Referencing: Universal Unique ID (UUID)

Another mechanism to track the same record across multiple applications Instead of keeping track of the system IDs., UCM publishes a UUID to the operational application The UUID is not exposed in the user interface Is stored in a column of the S_PARTY table Can be exposed using Siebel Tools Many customers use cross-referencing and the UUID at the same time

17

Data Quality Options

Data quality consists of 1. Data Cleansing 2. Data Matching(De-Duplication) Data Cleansing 1. Address validation and standardization 2. Capitalization 3. Abbreviations Data Matching identifies possible duplicate records for account,contact and prospect records Identify duplicates and merge them based on certain key attributes. E.g. from our previous case, if we decide that the key attributes of a unique customer are : first and last names and say the passport number. So, assuming we have the passport number stored in both the .NET and J2EE application, data matching in UCM will be able to identify that customer # 118

How to configure Data Quality

To configure data quality : 1. Enable the data quality component group 2. Set data quality parameters 3. Specify data quality settings 4. Specify data quality settings: Site Map > Administration Data quality > Data quality settings During integration cycle disable deduplication to simplify the environment.

19

What to Do with Duplicate Records

Once suspect duplicate have been identified, the records can be evaluated and: Merged with existing records Introduced as new UCM records (promoted) For records that are merged, UCM: Stores the duplicate record, with Merged as the type, and the surviving record, with History as the type, in the Source Data History table Merges parent records using survivorship rules to generate a new merged surviving parent record Updates the parent surviving record with the new merged surviving record Deletes the duplicate record
20

Survivorship Rules
Customer data originates in multiple systems Conflicts arise when different sources have different data for the same attribute For eg: two systems have different addresses for customer The role of UCM is to resolve these conflicts, which ensures that the best version record survives Survivorship rules are an automated mechanism for consolidating customer data to produce a best-version record Evaluates updates from each registered system Applies rules to determine which is the best value for each attribute Can be constructed by a system integrator or a CDI data steward Three comparison methods Recent

21

Survivorship Rules and Match Scores

UCM allows two thresholds to be set depending on the match scores from the matching process
Records with high match scores can be auto-merged, applying survivorship rules

High Score - 90 Records with match scores in the middle will be brought to the attention of the data steward for manual review and resolution Low Score - 70 Records with low match scores will be left as separate records

22

Setting Match Thresholds

Auto-threshold: Determines the high matching score threshold Account Auto Threshold Contact Auto Threshold Manual threshold: Determines the low matching score threshold Account Manual Threshold Contact Manual Threshold Can be constructed by system integrator or CDI data steward Can be set by a system integrator or a CDI steward

23

Survivorship Rule Structure

Employ comparison criteria to determine if a new value for a given attribute coming from any source is better than the existing value in the best-version record Three comparison methods: Recent History Source Multiple rules can be constructed for each object (account or contact) but only one rule can be active at a time Use attribute groups A group of fields that survive or get overwritten together Makes survivorship rule creation less cumbersome Each field within an object can only be included in a single attribute group To avoid survivorship rule conflicts
24

Building Survivorship Rule

Three primary steps: Create an attribute group for the survivorship rule Create the survivorship rule logic Activate the survivorship rule

25

Setting Survivorship Rules

26

Unmerge
For records that have been merged Surviving record--- record that remains after the merge Victim record ----- record that is merged and ceases to exist If merged records are unmerged after changes have been made to be the merged (surviving) record All changes and updates since the merge will be applied to the surviving record The victim record will be resurrected from the Source Data history table Merge Unmerge
A A2

A1 B

A2 B

Changed to merged record

27

Outbound Data Flow: Publish and Subscribe

Publish and subscribe allows records inserted or updated in UCM to be published to external systems Publication can be targeted to any system registered with UCM The mechanism for publication is a Siebel Workflow Workflow policies track changes to UCM records and flag them for production Then Workflow processes collect the appropriate information and guarantee publication of changes to specified (subscribed) systems Publish and subscribe requires that publication parameters be specified Can be real-time or daily-batch
28

Module 2: Overview Of Application Integration

29

Application Integration and the UCM SDH tables

UCM integration extends the EAI toolset by transferring data to SDH tables prior to placing data into Siebel base tables SDH tables provide a queuing station for processing incoming records Records can be analyzed for future processing Data must be processed at this layer before entering standard Siebel base tables This is the source of integration process differences between UCM and the standard Siebel CRM Enterprise product

30

Data Transfers to SDH tables

Data transfers to SDH tables use traditional methods Data layer data loads via the EIM Business layer data loads via standard integration application procedures Data transfers paths to SDH tables analogous to standard CRM case No data cleansing or matching
Business Layer Data Loads SDH Tables Base Tables

Data Layer Data Loads

31

Data Transfers to Siebel Base Tables

Are independent of how data is placed on SDH tables Data layer Business logic layer During this transfer(insert) data quality is activated Cleansing and matching Application of survivorship rules Implemented as a UCM workflow process The UCM Batch Manager

32

UCM Service Requests

External Systems request services from a UCM application Insert Query Update Delete UCM must determine which external system is requesting the service To validate privileges The System ID must be supplied in addition to the traditional inputs even if the UCM application is not physically connected to an external operational system

33

UCM External System Parameters

Request for service can only be accepted from a known source Registered system System ID System Name The records to be operated on also have UCM specific information associated with them Unique external record ID Unique user ID Transfer code describing data transfer mechanism

34

Inbound Data flow for the business layer

Request for UCM application services Submitted through a (CRMML) input file Security service checks privileges of requesting external system Transaction service executes operations
Siebel Adapter Through Dispatch Service To SDH tables Security and Transaction Services

Input data (CRMML)

Outbound Data

35

CRMML Input Files

Contain Information about external system --- system ID Service being requested Methods and arguments to be used User does not have to build the integration objects Internal integration objects provided by the UCM application The CRMML file directs the UCM application on which the integration object to use User does not have to specify UCM specific record information The UCM application will make these assignments when placing records on SDH tables

36

Security and Transaction Services

The security service checks the system ID specified in the input (CRMML) file for Insert ,Update, Query and Delete privileges If the system is not registered data will not be processed Transaction manager will perform the requested operations Uses an internal UCM integration object to obtain the data necessary to process a request

37

Implementing Inbound Data Flow at the Business Layer

Inbound Requests are processed by a workflow process HTTP Queue Based File Transport Protocol specified when system is registered
Siebel Adapter Through Dispatch Service To SDH tables Security and Transaction Services

Input Data (CRMML) Server Party Package

38

Outbound Data

Inbound/Outbound Processing
UCM has automated much of the inbound processing --- a user must: Prepare input(CRMML) file Run (appropriate) Server Party Package workflow to place records of SDH tables Run UCM Batch workflow to activate data quality and place data on Siebel base tables The UCM application supports an outbound publish and subscribe facility Insert and Update records can be published to external system subscribers
Individually or as a group In batch or real time mode
39

Module 3: Initial Data Loads into UCM

40

Populating UCM Data

Use business layer imports to enter , update or delete small amounts of data in the base tables Use the EIM server task to populate the SDH tables ( through interface tables) Use the UCM batch manager to transfer data from SDH tables to Siebel Base tables Non-Siebel Cleanse and match UCM Database
loading tool

External data may be cleansed

Interface table
41

UCM SDH tables

Siebel Base tables

Staging Tables
The UCM application uses two sets of staging tables before making data available to the Business Logic layer through Siebel Base tables The EIM staging tables: Data is staged to be transformed into a Siebel standard format The Source ,Data and history tables: Data is staged for cleansing and matching

42

Data Quality Options for Initial Loads

Data may be cleansed and matched while being transformed from the source , data and history tables or preprocessed before loading into EIM tables Cleansing and matching during data import adds loading latency ---time overhead If the Siebel Universal Connector is being used for cleansing and/or matching , it will need to be integrated into the application

43

Siebel Base Tables

User data is stored in one or more Siebel base tables Relationships between base tables (referential integrity) rely upon primary and foreign keys based on ROW_IDs ROW_ID is the system primary key for every base table ROW_ID is a system-generated value
PK PK FK FK

S_BU

S_ORG_TXT

S_ADDR_ORG

44

User Keys
Based on multiple columns, user keys are used to uniquely identify a row for EIM Primary and foreign keys based on ROW_ID are used for systemwide referential integrity
PK ROW_ID User Key FK BU_ID

Name

LOC

S_ORG_EXT

S_UCM_BU
45

UCM Data Loads

Loading Data into the EIM Tables Loading Data from the EIM Table to the SDH Tables Loading Data from the SDH Table to the Base Tables

46

Interface Tables
Staging area for data to be imported, updated, or merged into Source Data and History (SDH) Staging area for data imported by EIM EIM tables do not have ROW_Ids assigned
External Database

S_UCM_ORG_EXT

EIM_UCM_ORG

Interface Table External Data SDH Tables

47

S_UCM_CONTACT

Populate Interface Tables

Populate source data into interface tables using a non-Siebel process Use database vendor utilities to load data from flat files Use SELECT INTO to pull data from other SQL databases Write programs or stored procedures to cleanse or preprocess and insert rows Use direct SQL insert, update, or delete statements to update columns in EIM tables
External Data EIM_UCM_ORG

Non-Siebel process

48

Required Interface Table Columns

In addition to standard non-nullable columns IF_ROW_BATCH_NUM ---- Batch number IF_ROW_STAT --- Status ROW_ID Row ID The UCM application also requires a non-null inputs UCM_UID A unique UCM record identifier UCM_TYPE_CD --- SDH type (example : batch for initial loads) EXT_SYSTEM_NUM External System ID UCM_EXT_ID --- A unique external record identifier (usually assigned by the external system)

49

Source History Data (SDH) and EIM Schema

UCM Source Data History and EIM Tables
S_UCM_ORG_EXT S_UCM_CONTACT S_UCM_ORG_CHLD S_UCM_CON_CHLD EIM_UCM_ORG EIM_UCM_CON EIM_UCM_ORG_CHLD EIM_UCM_CON_CHLD Source and History table for accounts Source and History table for contacts Source and History table for child objects of account Source and History table for child objects of contact EIM interface table for S_UCM_ORG_EXT EIM interface table for S_UCM_CONTACT EIM interface table for S_UCM_ORG_CHLD EIM interface table for S_UCM_CON_CHLD

50

Referential Integrity
User key column mappings in interface tables are used to resolve ROW_Ids for SDH tables, maintaining referential integrity
User key PK(ROW_ID)

Acme

1-8D

Interface Table 1-8D

FK(ROW_ID)

UCM SDH Tables

51

Enterprise Integration Manager (EIM)

Server task that manages the exchange of data between Interface tables and Source Data History tables
S_UCM_ORG_EXT

EIM_UCM_ORG

S_UCM_CONTACT

External Data

52

Running the Enterprise Integration Manager

The EIM server task is run as a workflow from Administration Server > Jobs

53

Run as EIM Task with Parameter Settings

Creates an operating system (OS) process that writes log entries, reads an .ifb file, loads a dictionary and executes EIM steps

54

Configuration (.ifb) file

Role for initial data loads Specifies the batch , order and type of processing desired Should be placed in c:\Susrvr\siebsrvr\ADMIN For imports [Siebel Interface Manager] PROCESS = IMPORT EIM_UCM_CON [IMPORT EIM_UCM_CON] TYPE = IMPORT TABLE = EIM_UCM_CON BATCH = 100 ONLY BASE TABLE = S_UCM_CONTACT

55

Use Log Information to Troubleshoot

Navigate to Administration Server > Tasks Error information in Task view or log file

56

Verify Data Load into SDH Tables

Navigate to Administration Universal Customer Master > Contact (Account) > Source Data and inspect the Source Data table

57

UCM Batch Manager

Transfers data from SDH tables to Siebel Base tables Implemented by separate workflows for Accounts and Contacts UCM Account Batch Data Management workflow UCM Contact Batch Data Management workflow
S_UCM_ORG_EXT S_ORG_EXT

SDH table

Base Table

SDH table

Base column

58

Running the UCM Batch Manager

The UCM Batch Manager is run as a workflow from AdministrationServer > Jobs

59

Verify the Data Load into Siebel Base Tables

Navigate to Site Map > Contacts (Companies) > Contacts(Accounts) List and inspect the list

60

Determining Data Mapping

Need to analyze and determine : Which Siebel base table columns will store external source data Which interface (EIM) tables and Source Data History (SDH) tables will be used to import from source to destination
Source (External Where data ends up How to get there Destination (base)

Interface (EIM)

SDH

61

How to Get There

After verifying and documenting the source-to-base data mappings, determine EIM-to-SDH and SDH-base mappings
Source Entity Customer Customer Source Attribute Name Phone Address EIM Table and Column SDH Base Table Table and Column
S_ORG_EXT S_ORG_EXT S_ADDR_ORG S_ADDR_ORG S_ADDR_ORG

Base Column

NAME MAIN_PH_NUM ADDR CITY ZIPCODE

Interface (EIM)
62

SDH (UCM)

Table Column Mappings

Can be determined using Siebel Tools EIM Interface tables Source Data History Tables Source Data History Tables Base tables

63

Module 4: The UCM Batch Workflow and SDH Tables

64

The UCM Batch Manager and SDH Tables

This module will discuss two aspects of UCM Batch Manager operations on SDH tables. Data transfer to Siebel Base Tables Data transfer for child objects

65

The UCM Batch Manager

Batch processes insert requests queued on the Source, Data ,and History (SDH) tables Initial data loads (legacy systems) Incremental inserts Moves records from SDH tables to Siebel base tables Example: S_UCM_CONTACT S_CONTACT Two flavors of batch workflows for accounts and contacts UCM Account Batch Data Management Workflow UCM Contact Batch Data Management Workflow

66

UCM Account Batch Data Management Workflow

Start

Query by Id

Get Error Code

Error?

Process NoMatch

CDM Cleanse

CDM Match

Match id?

Process AutoMatch Process Review

Error?

Converter

PubSub

End

67

UCM Batch Workflow Steps

Query By Id CDM Cleanse CDM Match Process NoMatch Process AutoMatch Process Review

68

Query By ID
Invokes the Query method of the EAI Siebel Adapter business service Reads an object to be inserted from a SDH table Outputs a Siebel message
Workflow Process Props SDH int Obj Name OutputintObjectName PrimaryRowId Object Id SiebeMessage Siebel Message

Query By Id

69

CDM Cleanse
Invokes the Cleanse method of the UCM Data Quality Manager business service Reads a Siebel Message containing object to be inserted from SDH table Outputs a cleansed XML representation of the original Siebel Workflow Process Message Props
Base int Obj Name IntObjectName

SiebeMessage Siebel Message XML Hierarchy Cleansed Prop

CDM Cleanse

70

CDM Match
Invokes the Match method of the UCM Data Quality Manager business service Reads an XML representation of a Siebel Message containing an object to be inserted from SDH table Output indicates if record is a duplicate and the auto-merge Workflow level Process Props
Cleansed Prop XML Hierarchy

Number of AutoMatch

NumAutoMatch

CDM Match

MatchingrowId Cleansed Prop

71

To Merge or Not to Merge

The workflow splits the output of the match process into three cases (CaseType) No match ---- insert record Potential match --- activate survivorship rule to determine if merge is appropriate Transfer record for manual resolution

72

Process NoMatch
Is activated in the event that the incoming record is not scored as a match Record is inserted Invokes the HandleMatchCase method of the UCM Batch Manager business service
Workflow Process Props Matching Id Object Type Turn On Survivorship Cleansed Prop Processed Prop XML Hierarchy XML Hierarchy CaseType ObjectType TurnOnSE

Process NoMatch

73

Process AutoMatch
Is activated in the event that the incoming record is scored as a match Record evaluated by survivorship rule --- is either merged or inserted Invokes the HandleMatchCase method of the UCM Batch Workflow Process Manager business service Props CaseType
Matching Id Object Type Turn On Survivorship Cleansed Prop Processed Prop MatchingRowId ObjectType TurnOnSE XML Hierarchy XML Hierarchy

Process AutoMatch

74

Process Review
Is activated in the event that the incoming record is scored as a match but not resolved by survivorship rule Record is posted in Duplicate (Accounts) view Invokes the HandleMatchCase method of the UCM Batch Manager business service
Workflow Process Props CaseType Object Type Cleansed Prop ObjectType XML Hierarchy

Process Review

Processed Prop

XML Hierarchy

75

Post Process Activities

In the event that the record is resolved through an insert or a merge A real-time message can be published In the event record is queued for manual resolution The workflow is terminated and no information is published(in real--time mode)

76

Address for Contacts

Contact business objects come preconfigured with address child busniess components

77

The UCM Data Model and Child components

All UCM child components for each business object (Account and Contact) are stored in one table The explicit component object type is configured in each UCM business component to distinguish the child components
Busniess Layer Field BC BO

Data Layer

SDH Column

SDH table

78

Address for Contacts

Contact business objects come preconfigured with address child business components
Primary Business Component UCM Contact SDH UCM Contact SDH

UCM Contact Address SDH Business Object

Business Component

S_UCM_CONTACT

S_UCM_CON_CHILD Table

79

Source Data and History : Data schema

Provide the infrastructure necessary to support child structures Example : addresses for contacts
S_ORG_EXT Base table S_CONTACT

S_UCM_ORG_EXT

Source data History Table

S_UCM_CONTACT

S_CIF_EXT_SYST

S_UCM_ORG_CHLD

External System information table

S_UCM_CON_CHLD

Source Data History Child table

80

Creating UCM SHD Child Business Components

The UCM data model can easily be extended to include custom SDH components

81

Example: Address for Contacts

The UCM SDH Contact Address business component is UCM Contact Address Source Data and History

82

Custom Child Business Components

Name Business Components and Tables Assign Data Fields Configure UCM Fields Configure UCM Batch Workflow

83

Name Business Components and Tables

Copy the Siebel base business component and rename it Siebel suggested standard child business component names: UCM + <Parent BC Name> + <Child BC Name> + SDH

Copy of UCM Contact Address Source Data and History

84

Assign Data Fields

Change the predefined column names (ATTRIB_N) for the business component to the UCM SDH table Name field

85

Configure UCM Fields

There are five UCM specific fields that must be configured Component Field Name Object Name Column OBJECT_NAME Predefault Value Party Object name (Account/Conta ct) UCM Party base table Equal to child name Parent Row Id Field identifier

Object Table Object Type Par Rpw Id UCM UId

OBJECT_TABLE OBJECT_TYPE PAR_ROW_ID UCM_UID

86

Configure UCM Batch Workflow

The OBJECT_TYPE is used for the search specification in the UCM Batch Workflow parameter list Search Specification = Object Type Example for Addresses: [Object Type] = address

87

Module 5: UCM Messaging and CRMML

88

CRMML
An XML based markup language for data exchange and remote business process invocation Used for communication with UCM Inbound requests (insert, query, update, delete) Responses Publishing messages
Request

Back-Office Application

UCM
Response Publication These exchanges use CRMML messages

89

Example: Insert Contact Profile

<CRMML> System requesting the <CRMHeader> info <FromSystem> <SystemInfo> <SystemId>System123</SystemId> <SystemName>External App</SystemName> </SystemInfo> </FromSystem> <ToSystem> </ToSystem> <MessageDescriptor> </ MessageDescriptor> </CRMHeader> Requested transaction, <CIFPartyPackage> or method <InsertContactProfile> <Contact> Method argument <FirstName>Test</FirstName> <LastName>Contact</LastName> . </Contact> <InsertContactProfile> </CIFPartyPackage> </CRMML>
90

CRMML and Integration Objects

Integration Objects Define the structure of the CRMML message Are used as templates to create memory-resident instances of data for processing by business services UCM provides integration objects for different parts of the CRMML message Integration Object Description Type Envelope Integration Object Dispatcher map integration object UCM integration objects Represents CRMML header structure Contains rule sets for mapping from method to integration objects Represent the method argument (Contact, Account, etc)
91

CRMML Header
Contains Information identifying sending and receiving systems Used by Universal Customer Master for security verification Message descriptor identifying the message <CRMHeader> <FromSystem> <SystemInfo> <SystemId>System123</SystemId> <SystemName>External App</SystemName> </SystemInfo> </FromSystem> <ToSystem> </ToSystem> <MessageDescriptor> </ MessageDescriptor>
92

CRMML Header : System Info

Used to describe both the sending and receiving systems <FromSystem> - sender <ToSystem> - receiver For inbound messages, FromSystem SystemId is matched against registered systems as part of UCMs security check
SystemId used for security verification

<Systeminfo> <SystemId>123-456-789</SystemId> <SystemRole>Insurance App</SystemRole> <SystemName>Client System</SystemName> </Systeminfo>

93

CRMML Header : Message Descriptor

Identifies the message MessageId : Unique ID for the message Timestamp: when message was issued TimeToLive: how long (after timestamp) message is valid TransactionScope : reserved for future use
<MessageDescriptor> <MessageId>1T3D2005083115:45:32</MessageId> <TransactionScope>any</TransactionScope> <Timestamp>Sunday</Timestamp> <TimeToLive>OneYear</TimeToLive> </MessageDescriptor>

94

CRMML Header : Summary

CRMHeader element has the following structure
<FromSystem> <SystemInfo>

<CRMMLHeader>

<ToSystem>

<SystemInfo>

<CRMML>

<MessageDescriptor>

<Body>

95

CRMML Header: Integration Object

The CIF Envelope Integration Object is used to process the CRMML Header

96

CRMML Body
Contains one or more message elements representing: Inbound requests from an external applications to insert, query, update, or delete a database object Outbound response to a requested transaction Outbound publication messages to be sent to an external subscribing application Each message element includes: Software package must be CIFPartyPackage Method --- the requested action Argument the argument to the action
<SoftwarePackage> <Method> <Argument> <CIFPartyPackage> <InsertContact> <Contact>

97

Mapping to Integration Objects

UCM methods and arguments are mapped to integration objects according to a set of mapping rules These mapping rules define: What methods are allowed What arguments those methods take
Methods contain Arguments,which contain Fields <CIFPartyPackage> <InsertContact> <Contact> <LastName> </LastName> <FirstName> </FirstName> . Integration Objects contain Integration Components, which contain Integration Component Fields CIFContact Contact FirstName LastName

98

Mapping to Integration Objects

Continued

The mapping rules are defined in user properties of a mapping integration object The default mapping object is CIFDispMap

Envelope

Package

Method

CRMML/CIFPartyPackage/InsertContact -> CRMML/CIFPartyPackage/InsertContact;a;b;CIFContactInsert;CIFContactInsertRs; IXML_Operation_ADD Integration Object for inbound requests Integration Object for outbound responses

99

CRMML Structure: Summary

Structure of a base CRMML message
<FromSystem> <SystemInfo>

<CRMHeader>

<ToSystem>

<SystemInfo>

<CRMML>

<MessageDescriptor>

<CIFPartyPackage>

<InsertContact>

<Contact>

Package

Method

Argument

100

Complex CRMML Messages

Arguments may contain sub-arguments For example: Contacts contain addresses These sub-arguments are represented by sub-elements in the CRMML message <Contact_
INSPersonalAddress>

Method <InsertContactAddress>

Argument <Contact>

Method <InsertContactAddress>

<Contact_ INSPersonalAddress>

<Contact_ INSPersonalAddress>

101

Example: Insert Contact Address

<CRMML> <CRMHeader> .</CRMHeader> <CIFPartyPackage> <InsertContactAddress> <Contact> <FirstName>Test</FirstName> <LastName>Contact</LastName> <ListOfContact_INSPersonalAddress> <Contact_INSPersonalAddress IsPrimaryMVG=Y> <INSPersonalStreetAddress>100 Main St.</ INSPersonalStreetAddress> <INSPersonalCity>York</INSPersonalCity> <INSPersonalState>AL</INSPersonalState> <INSPersonalPostalCode>78221</INSPersonalPostalCode> <INSPersonalCountry>USA</INSPersonalCountry> </Contact_INSPersonalAddress> <Contact_INSPersonalAddress IsPrimaryMVG=N> <INSPersonalStreetAddress>50 Center St.</INSPersonalStreetAddress> . </ListOfContact_INSPersonalAddress> </Contact> </InsertContactAddress> </CIFPartyPackage> </CRMML> 102

Module 6: UCM Server Party Package Workflows

103

Party Package Workflows

Handle incoming requests Check source system privileges Create memory-resident instances of data for processing by business services, based on integration object definitions Execute requested operation Return response to requesting system Publish (real-time) to subscribing systems

104

Party Package Workflows

continued Include Workflow Process Name UCM Server Party Package Workflow (File Transport) UCM Server Party Package Workflow (MQSeries Server Transport) UCM Server Party Package Workflow(MQSeries Server Transport Error Handling) Description Reads CRMML message from a file and writes response to a file (for testing) Used for HTTP and MQSeries message transports. Also for HTTP and MQSeries transports, Models error handling.

105

UCM Server Party Package Workflow (File Transport)

Workflow is composed of the following steps:
Start Read From File Convert to XML Hierarchy Dispatch Message Security Check Convert Request In

PubSub

Transaction Manager

Realtime Pub/Sub

Convert Request Out

End

Write To File

Convert to Hierarchy
106

Read From File

Invokes the Receive method of the EAI File Transport business service Reads contents of a file into string Input file should contain a valid CRMML message
WorkFlow Process Props Input File Name File name Read From File: EAI File Transport Receive Start

XML

<value>

Convert to XML Hierarchy

107

Convert to XML Hierarchy

Invokes the XMLDocToXMLHier method of the XML Hierarchy Converter business service Converts an XML string into an XML hierarchy object
WorkFlow Process Props XML <value> Convert to XML Hierarchy: XML Hierarchy Converter XMLDocToXMLHier Read From File

Core Req Prop

XML Hierarchy

Dispatch message

108

Dispatch Message
Invokes the DispatchMessage method of the UCM Dispatcher business service Verifies structure of CRMML message Acts information to message components that is used by downstream business services Convert to XML
WorkFlow Process Props Hierarchy

Core Req Prop

XML Hierarchy Dispatch Message: UCM Dispatcher DispatchMessage

Disp Req Prop

XML Hierarchy

Security Check

109

Dispatch Message

Continued

Uses two integration objects, specified as user properties of the business service, to represent the message
Integration object used to process body Integration object used to process header

User Property Name DispatcherMapNa me

Value CIFDispMap

XMLEnvIntObjectN CIF Envelope ame

110

CIF Envelope Integration Object

Defines the allowed structure of the CRMML message header.

111

Dispatcher Map Integration Object

Contains a set of rules to map methods to integration objects used to process those methods Each rule is a user property of the mapping integration object Name identifies the message content that must be matched Value specifies the information that should be added to the method when a mapping is found
Content to match in order to trigger this rule Location where information gets inserted CRMML/CIFPartyPackage/InsertContact -> CRMML/CIFPartyPackage/InsertContact; a;b;CIFContactInsert;CIFContactInsertRs;IXML_Operation_ADD

Reserved for future use

Integration Object for inbound requests

Integration Object for outbound responses

Operation for the business process

112

Dispatch Message
Sample message before Dispatcher additions

Continued

<CRMML> <CRMHeader> . </CRMHeader> <CIFPartyPackage> <InsertContactProfile> <Contact> . </Contact> </InsertContactProfile> </CIFPartyPackage> </CRMML
113

Dispatch Message
Sample message after Dispatcher additions

Continued

<CRMML SiebelFINSObjType=Envelope> <CRMHeader SiebelFINSObjType=Header> .. </CRMHeader>

<CIFPartyPackage SiebelFINSObjType=Body> <InsertContactProfile SiebelFINSOperation=IXMLOperation_ADD Content added SiebelFINSIntObjName=CIFContactProfileInsert

here

Additional content added by Dispatcher

SiebelFINSRespIntObjName=CIFContactProfileInsertRs Dispatcher SiebelFINSMapOut=b SiebelFINSMapin=a> adds these attribute <Contact> values which come from the triggered .. mapping rule </Contact> </InsertContactProfile> </CIFPartyPackage> <CRMML>
114

Security Check
Invokes the SecurityCheck method of the UCM Security Manager business service Verify the SystemId in the message header against system registrations Return an error if the system is not recognized For each method in the request, verify that the system has the privilege required to execute the operation Failed operations are removed from the request and Dispatch put into WorkFlow Process the Fail Security Req Prop property set
Props Message

Disp Req Prop Fail Security Req Prop Security Req Prop

XML Hierarchy FailSecurityHierarchy XML Hierarchy Security Check: UCM Security Manager Security Check

Security Check
115

Security Check
Operation Name IXMLOperation_ADD IXMLOperation_DELETE IXMLOperation_QUERY IXMLOperation_UPDATE Privilege Insert Delete Query Update

Continued

User properties map operation (inserted by the Dispatcher) to privileges (assigned to systems in system registration)

In order to perform this operation

Requesting system must be registered with this privilege

116

Convert Request In
Invokes the XMLPropSetToPropSet method of the UCM Converter business service Each method is converted into the format required by the Transaction Manager Requesting System information is extracted from the header to make it available to downstream business services Security WorkFlow Process Uses integration object, CIF Envelope Check
Props Security Req Prop CIF System prop Tran Req Prop XML Hierarchy CIFSystemContainer XML Hierarchy Convert Request In: UCM Converter XMLPropSetToPropSet

Transaction Manager
117

Convert Request In
Sample converted method
Before
<CIFPartyPackage.> <InsertContactProfile.> <Contact> <FirstName>Test</FirstName> <LastName>Contact</LastName> <Status>Active</Status> .. </Contact> </InsertContactProfile> </CIFPartyPackage>

Continued

After
<CIFPartyPackage.> <ListOfCIFContactProfileInsert.> <Contact> First_spcName=Test Last_spcName=Contact Status=Active .. /> </ListOfCIFContactProfileInsert> </CIFPartyPackage>

118

Transaction Manager
Invokes the Execute method of the UCM Transaction Manager business service Performs operations in message by invoking services configured in its user properties
WorkFlow Process Props Tran Req Prop Turn On Data Mgmt Turn On Survivorship Status Object Notification Req Prop Trans Resp Prop
XML Hierarchy XML Hierarchy TurnOnDM TurnOnSE StatusObject NotificationHierarchy

Convert Request In

Transaction Manager: UCM Transaction Manager Execute

Convert Request Out

119

Transaction Manager Operations

Supported Operations Operation Insert Update Description Insert into Source Data history tables with status,Queued. Insert into Source Data History tables with status,Update. Invoke Survivorship and Data Quality Manager, if enabled. Insert updated record directly into base tables. Query base tables. Locate record to delete based on user key in message. Store delete request in Source Data History table. Store historical best version. 120

Query Delete

Transaction Manager Configuration

Configuration Transaction Manager inputs in the workflow to Enable data management and survivorship TurnOnDM enable data management (default is false) TurnOnSE--- enable survivorship (default is false) Include a representation of the affected object in the response to the requesting system StatusObject (default is true)

121

Convert Request Out

Invokes the PropSetToXMLPropSet method of the UCM Converter business service Generates a response that combines information from its inputs Response will be converted to an XML hierarchy and then sent Transaction back to the requesting system
WorkFlow Process Props Tran Resp Prop Notification Req Prop CIF System Prop Fail Securtiy Req Prop Conv Resp Prop
XML Hierarchy NotificationHierarchy CIFSystemContainer FailSecurityHierarchy XML Hierarchy

Manager

Transaction Manager: UCM Transaction Manager Execute

Realtim e Pub/Su b
122

Considerations
Error Handling Configuring Inbound Server Communication

123

Error Handling
The UCM Server Party Package Workflow (MQSeries Server Transport Error Handling) demonstrates how error handling could be added to a workflow.
Convert to XML Hierarchy Dispatch Message Security Check Convert Request In Transaction Manager Error Handler: UCM Converter: GenerateErrorPropSet Error Handler: UCM Converter: GenerateErrorPropSet

End

124

Error Handling

Continued

The UCM Converter service includes a method to aid with error handling

WorkFlow Process Props Error Code Error Message Disp Req Prop
ErrorCode ErrorMessageText XML Hierarchy

UCM Converter: GenerateErrorPropSet

Error Hierarchy Prop

XML Hierarchy

125

Configuring Inbound Server Communication

Your UCM environment must be configured for inbound server communication Configuration procedure depends on the system transport protocol MQSeries Transport HTTP Transport

126

Module 7: UCM Batch Data Management Workflows

127

Batch Data Management Workflows

Process data queued in Source Data and History tables During initial data load As a result of an insert request processed by Transaction Manager Separate workflows for contacts and accounts UCM Contact Batch Data Management Workflow UCM Account Batch Data Management Workflow Each workflow performs data cleansing, matching, and real-time publication (if enabled)

128

UCM Contact Batch Data Management Workflow Performs the following actions on a single input record
Cleansing Start Matching Real-Time publication Query by ID
CDM Cleanse Get Error Code
Error

CDM Match

match

Auto Match ? Yes No Process Review

Matching

Error?

Process NoMatch

Process AutoMatch

End

True Realtim ePub/S ub False

Converter

Publishing

PubSub

129

Data Cleansing
Performed by Cleanse method of the UCM Data Quality Manager business service Invokes third-party data cleansing software (if enabled) Output is cleansed record
Start Retrieve record data for processing

Query by ID CDM Cleanse

Perform data cleansing

130

Data Matching (Deduplication)

Performed by Match method of the UCM Data Quality Manager business service Three possible outcomes: No match found Match found that must be reviewed Match found that can be processed automatically
Error match

Get Error Code

CDM Match Yes

Auto Match ?

No Process Review

Error?

Process NoMatch

Process AutoMatch

131

Data Matching
Continued All three match cases are processed by the same business service method: UCM Batch Manager HandleMatchCase

Process NoMatch

Process AutoMatch

Process Review

132

Data Matching
Continued Thresshold settings determine how match is handled
Process NoMatch High Score - 90 Process AutoMatch Records with match scores in the middle will be brought to the attention of the data steward for manual review and resolution. Records with high match scores can be automerged,applying survivorship rules.

Low Score - 70 Process Review Records with low match scores will be left as seperate records.

133

Adjusting Threshold Settings

Configure user properties of the UCM Batch Manager business service to adjust merging thresholds User Property Default Value Account Auto Threshold Account Manual Threshold Contact Auto Threshold Contact Manual Threshold 90 70 90 70

134

Publishing
Data changes can be published to subscribing systems Result of auto merging of records Result of insert of record (no match condition) Publishing is addressed in the next module
Process NoMatch Process AutoMatch

Realtim ePub/S ub False

True

Converter

Convert data to format for publishing Publish Transaction

PubSub

End

135

Running Batch Data Management Workflows

Create a Job Configure Job Parameters Run the Job

136

1. Create a Job
In the Administration Server Management > Jobs view. Create a new job to run the Workflow Process Batch Manager component/job

Select this component/job

If job is repeating specify repeat info in job details

137

2. Configure Job Parameters

Include Search Specification: Identifies items to process Workflow Process Name : Identifies workflow to invoke on each item

138

3. Run the Job

Once the job parameters are set, run job
Click start button to start the job

139

Module 8: Publish and Subscribe

140

Publish and Subscribe

Allows records inserted or updated in UCM to be published to external systems Distributes the best version customer profile Systems registered as subscribers receive publication Update: Contact123 Subscriber 1 messages
UCM
Update: Contact123

Subscriber 2

Update: Contact123

Subscriber 3

141

Publish/Subscribe Modes
Publish and Subscribe framework supports two operative modes: Real-time Transaction are published as they happen Each individual transaction is published to real-time subscribers Daily Batch Mode Transactions are published once per day Best version of record is published, individual transactions are not Systems can mix and match modes Contact updates received in real-time Account updates received from daily batch job
142

Publication Messages
Publication messages are CRMML messages FromSystem is UCM ToSystem is subscriber Message body content is constructed using integration objects Real-time publishing uses the response integration object associated with the particular UCM transaction Batch publishing uses the integration object type specified as input to the both Publish/Subscribe business service

143

Workflows and Real-Time Publishing

UCM Workflows include steps to perform near real-time publishing Decision step invokes publishing method if publishing is enabled Process
Transaction Manager response is published PubSub PubSub Realtim ePub/S ub Convert to Hierarchy End UCM Server Party Package Workflow
144

NoMatch Process AutoMatch Realtim ePub/S ub Convert Request Out Converter Result of match is published

PubSub

UCM Contact/Account Batch Data Management Workflow

Real-Time Publishing of Insert Operations

Insert operations are processed in two steps: Queue insert in SDH tables Process insert, performing any necessary data cleansing and matching Both steps will generate a publish message The subscriber system should only process the second message Subscriber should ignore publish messages reporting that an insert operation has been queued.

145

Workflows and Batch Publishing

UCM Daily Publish workflow supports regular batch publishing

Start

UCM Daily Publish

End

146

UCM Publish/Subscribe Business Service

Workflows invoke this business service to perform publishing Method PublishMethod Inputs XML Hierarchy Contains data to be published Integration Object Used to generate the publication message (batch publishing only)

147

Configuring Publishing: Real-Time

Set workflow user property to enable publishing

User Property names differ! Set to true to enable publishing

148

Configuring Publishing: Real-Time Continued

User Property names differ! Set to true to enable publishing

149

Configuring Publishing: Daily Batch

Configure a job to run the UCM Batch Publish Subscribe component This job will run the Daily Batch workflow

Integration object to use Number of records to processed in one batch

150

Subscribers
Registered systems can be configured as subscribers to receive publication messages in real-time or batch
URL where publication messages are sent

Required for batch publishing

Updated each time a message is published to this subscriber

151

Module 9: Wrap-Up

152

Course Goal
To enable participants to integrate Siebel Universal Customer Master (UCM) with external applications Legacy systems External databases Was achieved by covering the following topics Initial data loads into UCM at the data layer Incremental data loads into UCM at the business logic layer Managing outbound data flows via publish and subscribe The CRMML messaging protocol for requesting UCM services

153

Initial Data Loads into UCM at the Data Layer

Batched data layer loads into UCM are a two-stage process Loading from Siebel Interface table to Source Data and History (SDH) tables using the normal Siebel EIM process Loading from SDH tables to Siebel base tables using UCM Batch Manager Interface tables Are loaded using non-Siebel software Operated on by the Enterprise Integration Manager to transfer data to SHD tables The UCM Batch Manager is used to move data from SDH tables to Siebel base tables With data cleaning and matching to produce best version data Data is then accessible from the Siebel applications standard business logic displays 154

Data Loads into UCM at the Business Layer

Data loads at the business logic layer employ two distinct process managers: The Server Party Package Manager The UCM Batch Manager The Server Party Package Manager: Reads input data Identifies the data source Dispatches the incoming Insert message Queues the data on the SDH tables The UCM Batch Manager is used to move data from SDH tables to Siebel base tables With data cleansing and matching to produce best version data Data is then accessible from the Siebel applications standard business logic displays
155

Managing Outbound Data Flows

Records that have been inserted or updated can be selectively published to subscribers Making best version record available throughout the data ecosystem Real-time or batch mode Outbound publication service is performed by The Server Party Package Manager for Query, Update, and Delete operations The UCM Batch Manager for Insert Operations

156

The CRMML Messaging Protocol

XML based mark-up language used by UCM Inbound server requests Insert Query Update Delete Outbound server features Publish and Subscribe Formatted to provide Requesting external system ID Service(s) being requested Pointers to predefined UCM integration objects
157

Some Guidelines
Initial data for UCM should be cleaned outside (Before loading into UCM) EIM may be more effective to load data into UCM from external systems for performance reasons Batch publish method is more effective when send data from UCM to external systems UCM is not complete Customer data store, UCM can not store complete customer information. UCM Contact table has limited columns only; this data is primarily for referencing customer data between multiple external systems UUID is not numeric and big in size (about 20+ characters string), so it may not be useful as general purpose contact Id
158

Thank You

159