Professional Documents
Culture Documents
Contents
Applicability, Goals and Requirements _________________________________________________________3 1.1 1.2 1.3 2 Goal of Using this Service ___________________________________________________________3 Staff and Skills Requirements _______________________________________________________3 Example of a Business Scenario ______________________________________________________3
Best Practice Procedure and Verification ___________________________________________________6 2.1 Preliminary Tasks _________________________________________________________________6 2.1.1 Monitoring Concepts____________________________________________________________6 2.2 Procedures ______________________________________________________________________10 2.2.1 R/3 Back-End System Monitoring ________________________________________________10 2.2.2 CRM Server Monitoring ________________________________________________________15 2.2.3 Field Sales Specific Monitoring __________________________________________________23 2.2.3.1 CRM Server________________________________________________________________23 2.2.3.2 Communication Station _______________________________________________________25 2.2.3.3 Mobile Client Monitoring _____________________________________________________28 2.2.4 Interaction Center Specific Monitoring_____________________________________________29 2.2.4.1 CRM Server________________________________________________________________29 2.3 Verification _____________________________________________________________________31 2.3.1 Middleware Portal SMWP _____________________________________________________32 2.3.2 Monitoring CRM Outbound Queues SMQ1________________________________________33 2.3.3 QOUT Scheduler SMQS ______________________________________________________36 2.3.4 Monitoring CRM Inbound Queues SMQ2 _________________________________________37 2.3.5 Scheduler Status SMQR _______________________________________________________39 2.3.6 TRFC Monitoring SM58 ______________________________________________________40 2.3.7 Monitoring the Middleware Message Flow Statistics SMWMFLOW ____________________41 2.3.8 Check Flow Definitions SMO8FD _______________________________________________42 2.3.9 Middleware Trace _____________________________________________________________43 2.3.10 Monitor Request R3AR3 ______________________________________________________43 2.3.11 Monitor Initial Load R3AM1 ___________________________________________________43 2.3.12 Monitoring the Replication & Realignment Queues SMOHQUEUE ____________________44 2.3.13 Queue Information for CRM Mobile Sites SMWMQUEUES __________________________45 2.3.14 Communication Monitor SMWMCOMM _________________________________________46 2.3.15 Message Recovery CMWQ ____________________________________________________47 2.3.16 DCOM Connector Monitor ______________________________________________________47 2.3.17 Communication Station Log File: TransferService.Log ________________________________48 2.3.18 Windows Performance Monitor __________________________________________________49 2.3.19 SAP Connect Monitor SCOT ___________________________________________________50 2.3.20 SAP Phone Administration SPHB _______________________________________________51
2002 SAP AG
Best Practice: mySAP CRM Field Sales Monitoring 2.3.21 2.3.22 2.3.23 2.3.24 2.3.25 2.3.26 2.3.27 2.3.28 2.3.29 2.3.30 2.3.31 Internet Communication Manager Monitor SMICM _________________________________52 Gateway Monitor SMGW _____________________________________________________53 R/3 Buffer Monitor ST02 ______________________________________________________53 Workload Monitor ST03N _____________________________________________________53 Database Performance Analysis ST04 ____________________________________________54 Operating System Monitor ST06 ________________________________________________55 Local and System-Wide Work Process Overview SM50 / SM66 _______________________55 Display Statistical Records STAD _______________________________________________56 ABAP Dump Analysis ST22 ___________________________________________________56 System Log SM21 ___________________________________________________________57 Update Error Monitor SM13 ___________________________________________________58
2.4 Troubleshooting__________________________________________________________________59 2.4.1 Problems during Data Exchange __________________________________________________59 2.4.1.1 Error Detection in Delta Download______________________________________________59 2.4.1.2 Error Detection in Initial Download _____________________________________________65 2.4.1.3 Error Detection in Upload _____________________________________________________68 2.4.2 Problems with E-Mail Sending (Outbound Direction) _________________________________71 2.4.3 CRM Server Performance problems or high I/O load due to excessive traces and logs being written 72 2.4.4 Mass changes of data (creating, changing) on the OLTP system leads to reduced system performance__________________________________________________________________________74 2.4.5 Performance problems due to statistics updates on tRFC/qRFC tables_____________________74 2.4.6 System performance degrades as the size of the tRFC/qRFC tables increases _______________74 2.4.7 Problem situation 1 ____________________________________________________________75 2.4.8 Problem situation 2 ____________________________________________________________75 2.4.9 Problem situation 3 ____________________________________________________________75 2.4.10 Problem situation 4 ____________________________________________________________75 2 ______________________________________________________________________________________77 Feedback and Questions ________________________________________________________________77
2002 SAP AG
The general monitoring goals are: Detect documents in an error status Refer to reliable procedures for error handling
Measure performance of data exchanges Monitoring objects and their attributes are usually displayed in the alert monitors. However, some interface points must be checked using generic tools and procedures. These are also described in this document.
2002 SAP AG
Best Practice: mySAP CRM Field Sales Monitoring 7. The CRM Server updates the CRM Online database and after replication to the CDB puts the information to update the mobile clients database into the clients outbound queue on the CRM Server. 8. The mobile client picks up its data from the CRM Server.
SAP CRM
SAP R/3
IPC
Configure product
IPC
Technical Process Flow Overview The business process described above is technically represented by the message flow used for transferring data from the mobile client to the CRM Server and to the R/3 back-end system. The flow is based on the standard SAP business process steps of the Standard Sales Order Processing Scenario. This general description can be applied to most communications between the mobile clients and the R/3 back-end and/or CRM Servers. Sales orders are represented by the corresponding BDoc message type being transferred between the mobile client and the CRM Server. BDoc types can also transfer all types of data between the mobile client and the CRM Server (such as system information and administrative information) and then to all other receivers. Message Flow Within CRM In all CRM scenarios, message flow is used to supply all components of the CRM landscape with necessary information in accordance with subscription rules. The figure below shows message flow for CRM Field Sales scenario.
2002 SAP AG
R/3 Adapter
R/3 External interfaces (XML - SOAP, IDoc) Both XML/SOAP and IDocs can be generated from the ABAP complex data type used to define the external interface
Mobile Client
Sync BDoc Flow
XIF Adapter
XIF Adapter R/3 Adapter Mobile Bridge: Msg BDoc -> Sync BDoc
R/3
Mobile Client
CRM Middleware Adapters and Services The CRM Adapter is called by the message flow to pass inbound BDoc messages to the CRM Server application for validation. In case of a successful validation by the CRM Server applications, the content of the BDoc message is written (from the extension part) to the corresponding tables of the CRM database. If the validation was not successful, the BDoc message is sent back to the sender. The Replication and Realignment Service determines whether a replication and/or a realignment is necessary or not. If a realignment has to be performed for a BDoc message, this message is copied into a separate realignment queue for further processing. If realignment is not required, the receiving sites for a BDoc message are determined. The Mapping Service maps synchronization BDoc types to messaging BDoc types. The reverse direction is mapped using the Mobile Bridge. The Mobile Bridge takes a messaging BDoc type and creates one or more synchronization BDoc types (1:n relationship). It also takes one or more synchronization BDoc message and produces exactly one messaging BDoc message of one predefined type (n:1 relationship). The CDB Service saves the content of a synchronization BDoc message in the corresponding CDB tables.
Note: Mobile Bridge and CDB Service are only active in Field Sales (=MSA) or Field Service (=MSE) Scenarios. (CDB = Consolidated Database on the CRM Server, MTS = Message Transfer Service. MTS client is on the Mobile Client, MTS server - on the Communication Station)
2002 SAP AG
Detected errors
Actions
Automatically generated
Undetected errors
Middleware Portal Display BDoc Messages Monitoring of queued RFC Middleware Trace RFC log files
The following table briefly lists the Software Components and the monitoring functions associated with them:
2002 SAP AG
Server Types
Mobile Client
Monitoring Function
1. Monitor the outbound queue of the mobile client 2. Monitor the inbound queue of the mobile client 3. Monitor database 4. Monitor the operating system 5. Monitor network throughput 6. Message Recovery
Scenario
Only MSA/ MSE
Communicatio n Station
1. Monitor message flow from all mobile clients to the CRM Server 2. Monitor network throughput 3. Monitor operating system 1. Monitor outbound queue 2. Monitor R/3 Basis System 3. Monitor database 4. Monitor operating system 5. Monitor network throughput
CRM Server
1. Start with the Middleware Portal. In case of an error go to the appropriate transactions. 2. Monitor outbound queue, data transfer from the CRM Server to mobile clients 3. Monitor inbound queue, data transfer from mobile clients to the CRM Server 4. Monitor outbound queue, data transfer from the CRM Server to the R/3 back-end system 5. Monitor inbound queue, data transfer from the R/3 back-end system to the CRM Server 6. Monitor replication and realignment queues (only MSA/ MSE) 7. Monitor Middleware message flow 8. Monitor flow control 9. Monitor the Middleware log 10. Monitor status of BDoc types 11. Monitor initial/delta load / requests from the R/3 back-end system to the CRM or from the CRM to the CDB (only MSA/ MSE) In addition to the Middleware monitors, the SAP basis has to be monitored: 1. Monitor R/3 Basis System 2. Monitor SAP Connect interface for mailing and/or fax (if used in CRM Online or Interaction Center Scenarios) 3. Monitor CTI interface (if used in Interaction Center scenario) 4. Monitor database
2002 SAP AG
Server Types
Monitoring Function
5. Monitor operating system 6. Monitor network throughput
Scenario
2002 SAP AG
SMQ2
Mobile Client(s)
Inbound Business Processing Queue Outbound Queue DB & Log
Comm. Station
CRM MW System
Outbound Queue
R/3 System
Inbound Queue Outbound Queue SAP Business Processing
DCOM Connector
Log file
Flow / Process Control Engine & other services R/3 Application Database CDB
SMQ1 / SMQS ST03 ST06 SM58 ST22 DB02 SM50 / SM66 SMGW
Non CRM-specific SAP System monitoring transactions: ST02, ST04, SM21, SMGW, SM13, STAD 2002 SAP AG 9
2.2 Procedures
2.2.1 R/3 Back-End System Monitoring
After the CRM Server has been installed and the initial load has been successfully performed, system load (data exchange) to and from the R/3 back-end system can be categorized as follows: R/3 Back-End System CRM Server Delta load (business data and conditions) Request of specific data Synchronization load (customizing data only) Regular communications using RFC calls
CRM Server R/3 Back-End System Delta uploads from CRM via outbound adapter Regular communications using RFC calls
The following diagram shows the main points of interface on the R/3 back-end system. The transactions used to monitor these interface points are listed along with a brief label describing their function.
2002 SAP AG
10
Mobile Client(s)
Inbound Business Processing Queue Outbound Queue DB & Log
Comm. Station
CRM MW System
Outbound Queue
R/3 System
Inbound Queue Outbound Queue SAP Business Processing
DCOM Connector
Log file
Flow / Process Control Engine & other services R/3 Application Database CDB
SMQ1 / SMQS ST03N ST06 SM58 ST22 DB02 SM50 / SM66 SMGW
2002 SAP AG
11
Best Practice: mySAP CRM Field Sales Monitoring The table below lists transactions for monitoring the R/3 back-end system: Middleware Specific Monitoring R/3 Back-End System Monitoring Queue Monitoring Activities / Functions qRFC Outbound Queue Monitor: Monitor data exchange from the R/3 back-end system to the CRM Server Queues should be relatively short and quickly processed Check, if the qRFC Version is up to date (see SAP Note 438015) Transaction Code Monitoring Frequency Periods and Events Several times a day depending on the business process In case of an error message During mass updates Use Alert Monitoring
SMQ1
To prevent data inconsistencies, you need to monitor the interfaces regularly for aborted or stopped data transfer Status of Queue Scheduler Monitor status of the QOUT Scheduler
SMQS
Basis Monitoring: SAP Performance Monitors R/3 Back-End System General Monitoring Activities / Functions R/3 System Buffer Monitor: Monitor memory resource usage for specific R/3 application servers Implement parameter recommendations for memory management per EarlyWatch analysis Use normal R/3 system monitoring practices ST03N Daily In case of performance problems Monitor RFC response time statistics for CRM Monitor the Dialog response time for online transactions ST04 Daily In case of performance problems Monitor database statistics Monitor the buffer cache hit ratio Transaction Code ST02 Monitoring Frequency Periods and Events Weekly In case of performance problems
2002 SAP AG
12
R/3 Back-End System General Monitoring Activities / Functions Database Performance Monitor
Monitor the growing of tables and indices, especially on tRFC- and qRFC-tables
Operating System Monitor: Monitor hardware load during high RFC transmission times
ST06
Several times a day depending on the business process In case of performance problems
Basis Monitoring: General System Checks R/3 Back-End System General Monitoring Activities / Functions Local Work Process Overview: Monitor current state of individual work processes Ensure that enough work process capacity is available at peak times Transaction Code SM50 Monitoring Frequency Periods and Several times a day depending on the business process Daily In case of performance problems or error messages Several times a day depending on the business process In case of performance problems or error messages Several times a day depending on the business process In case of an error message Daily In case of an error message Daily In case of an error message In case of an error message
System-wide Work Process Overview: Similar to SM50 but for system-wide statistics
SM66
ST22
System Log: Check for general system errors SM13 SM21 STAD
2002 SAP AG
13
OBJNAME
2002 SAP AG
14
2002 SAP AG
15
Mobile Client(s)
Inbound Business Processing Queue Outbound Queue DB & Log
CRM MW System
Outbound Queue
R/3 System
Inbound Queue Outbound Queue SAP Business Processing
DCOM Connector
Log file
Flow / Process Control Engine & other services R/3 Application Database CDB
2002 SAP AG
16
The table below lists transactions used for monitoring the CRM Server: CRM Server Central Monitoring Activities / Functions Middleware Portal Monitoring the CRM Middleware Portal with transaction SMWP can be subdivided into the following areas: Generation Information containing: BDocs Types: Generation of structures BDocs Types: Generation of other runtime objects Replication objects Publications: Missing Indexes, Status of Delta/Initial Generation, Flow Definitions Runtime Information containing: Adapter Status Information Replication and Realignment Queues BDoc Messages in the Flow Monitoring Type SMWP Monitoring Frequency Periods and Events Several times a day depending on the business process After implementing transports
qRFC Outbound Queue Monitor: Monitor data transfer between the R/3 back-end and the CRM Server and between the CRM Server and mobile clients and other connected systems Queues destined for the R/3 back-end and other stady connected systems should be relatively short and quickly processed, queues to destination NONE too Entries destined for the mobile clients and BW remain in the queue until each receiver fetches its data When the queue entries for a client reach 10,000, the queue should be closely monitored (e.g. issue warning). When the entries reach 100,000, severe problems may occur and performance will be affected. Administrative actions must be taken in these cases, any deletion of queues causes data inconsistencies If a queue that is in use between a mobile client and the CRM Server is deleted, it will cause data inconsistency between the CRM
SMQ1
2002 SAP AG
17
CRM Server Central Monitoring Activities / Functions Server and the mobile client.
Monitoring Type
In severe cases when a client cannot be manually rebuilt, it can be brought back into a consistent state by rebuilding the client data from the CDB (AC_EXTRACT) SMQS
QOUT Scheduler
Daily In case of performance problems or error messages Several times a day depending on the business process
Ensure that all destinations are registered. Only destination NONE must be excluded. See also SAP Note 400330 SMQ2
qRFC Inbound Queue Monitor: Monitor data transfer between the R/3 back-end system and the CRM Server and between mobile clients and the CRM Server and all other queues, which have to be stored in the CRM Online DB Messages in the inbound queue are processed according to the capacity of the CRM Server. High number of entries in the inbound queue can indicate insufficient capacity on the CRM Server
QIN Scheduler Status This transaction runs the scheduler to check the inbound queues on the CRM Server Ensure that all inbound queues are registered (type R). Register them, if necessary, using the transaction SMQR If a queue is registered and has not been processed for a long time, the administrator has to check the reason for it. Possible a scheduler problems may exist. Check the maximum processing time of the inbound queues. Set the maximum processing time of a queue ("Max-time") to 60 seconds for all queues during normal operations
SMQR
SM58
2002 SAP AG
CRM Server Central Monitoring Activities / Functions (tRFCs) processed on the CRM Server (e.g. workflow) SM58 is linked to the Replication and Realignment queues in error status in the EXTRACT and/or AC_EXTRACT queue (click on the error symbol) Message Flow Statistics Collects statistical data about the workload on the CRM Server caused by BDocs messages Use this as a starting point for analyzing performance problems The message flow is a central function in the CRM Server and uses BDocs Display statistics of the message flow within the CRM Server Ensure that the middleware message flow statistics are switched on BDoc Messages / Summary Application error analysis SMW01: Display the data of a BDoc message and possible errors SMW02: Display BDocs message summary in dependency on the site ID Middleware Trace Developer Trace Summary of Unprocessed BDoc Messages Check Flow Definitions Only after changes in the customizing Consistency Check for Flow Definitions
Monitoring Type
SMW01 / SMW02
Several times a day depending on the business process In case of an error message
After BDoc type changes or changes in services or in the flow are made In case of an error message
2002 SAP AG
19
CRM Server R/3 Adapter / Load Monitoring Activities / Functions Monitor Load Status Checks, whether the initial load was successfully completed Object status and actions: RED: Refer to SAP Notes 429423 and 350176 for initial load problem analysis YELLOW: initial download must still be done / is running GREEN: OK
Monitoring Type
Monitoring Frequency Periods and Events In case of an error messages during/after initial load Depending on the object status
R3AM1
Monitor Request Used in exceptional cases to bring the database into a consistent state after a data lost for instance in the CDB
R3AR3
In case of an error message, if the databases are not consistent and a request from the R/3 back-end, the CRM or the CDB is necessary
Basis Monitoring: General System Checks CRM Server General Monitoring Monitoring Type Activities / Functions R/3 System Workload Analysis: Monitor RFC response time statistics for CRM Monitor the dialog response time for Online transactions (e.g. CIC0) SM50 Monitor current state of individual work processes Ensure that enough work process capacity is available at peak times SM66 ST03N
Hourly In case of an error message or performance problems Several times a day depending on the business process In case of an error message or performance problems Upon error Daily
System-wide Work Process Overview: Similar to SM50, but for system-wide statistics
Internet Communication Manager (ICM) Monitor Monitor current state of ICM and services
SMICM
2002 SAP AG
20
Monitoring Type
Buffer Monitor: Monitor memory resource usage for specific R/3 application servers Use normal SAP System monitoring practices
ST02
Weekly
Operating System Monitor: Monitor hardware load during high dialog user load / RFC transmission times
ST06
Several times a day depending on the business process Daily In case of an error message Daily In case of performance problems Daily In case of an error message Hourly In case of an error message In case of an error message
DB02
ST04
ST22
SM21
Gateway Monitor / Connection List: Monitor the active connections to other servers Display gateway trace Check the SAP return code and CPI-C return code values for errors
SMGW
SM13
STAD
Several times a day depending on the business process In case of an error message In case of an error message
2002 SAP AG
21
Description Consumer of adapter functionality Flag if application is activated Prefix for queue names in QRFC
The table SMOFPARSFA contains control parameters of the CRM Server. Other settings of the parameters depend on the business scenario you are using, see CRM Installation Guide. Current, experience-based parameter settings are checked during each SAP Service Session (EarlyWatch, GoingLive, etc.). The SAP recommendations should be applied according to the instructions given in the SAP Service report. The following parameters are important for CRM Field Sales and are provided here as an example: Parameter key CRMGENERAL CRMGENERAL CRMGENERAL RRS_COMMON RRS_COMMON FLOW Parname TRACE-LEVEL TRACE-LEVEL TRACE-LEVEL MAX_PACKAGE_SIZE RRQUEUE_PARALLEL USE_INQUEUE_ALWAYS Description ENV=G ENV=* ENV=I Note 453882 (only MSA/MSE) Note 453882 (only MSA/MSE) Note 529764 (for distributed systems)
2002 SAP AG
22
Best Practice: mySAP CRM Field Sales Monitoring BDoc Flow Statistics Activation Ensure that the middleware kernel application statistics are switched on. Call transaction SMWMFLOW. Execute Goto -> Activate Statistics. Select message flow statistics. Activate monitoring of the middleware message flow. To activate the application statistics, select Kernel application statistics, select the change mode, mark the field MW (Middleware Message Hub Statistics. Save. Communication Monitor Collector Activation For activating transaction SMWMCOMM, the Conntrans Performance monitoring, you have to choose Environment-> Run Collector
R/3 Adapter
qRFC Inbound
Synchronization Flow
XIF Adapter
CRM Adapter
Routing (Simple Replication)
XIF Adapter R/3 Adapter Mobile Bridge: Msg BDoc -> Sync BDoc
Outbound Adapter
qRFC Outbound
In this section, all CRM Middleware monitors specific for Mobile scenarios are described.
2002 SAP AG
23
Monitoring Activities / Functions Queue Information for CRM Client Sites Display all mobile sites defined in the CRM Server together with the name of the queue assigned to each of these sites
Monitoring Frequency Periods and Events Only for error tracking in seldom cases
Replication Queues Monitor Displays information about the status and contents of the replication and realignment queues in the clients defined in the CRM Server The EXTRACT queue shows to which clients a specific BDoc type will be distributed To check the status of the queues Deselect the flag: Display current client only Press the Refresh button Search for queues with Status = Hold number of entries > 0 and a flash icon button in the right-most column Correct the error situation for the indicated queues (error handling in SM58) Note: Do not press the trash icon button if it appears next to an entry. Communication monitor Communication monitor: monitoring individual sessions, statistics of the data exchange for each site
SMOHQUEUE
SMWMCOMM Middleware Monitoring Mobile Client Communication Monitor CMWQ Monitoring -> Mobile Client -> Message Recovery OS07
Mobile Client Message Recovery Unprocessed message recovery: reporting messages informing the CRM Server about errors during the import on the mobile clients Operating System / Gateway The SAP system statistic collector daemons, SAPOSCOL and RFCOSCOL, run on the Communication Station and gather hardware resource consumption data. Complementary programs run on the CRM Server and collect and display statistical data.
Daily
2002 SAP AG
24
Monitoring Activities / Functions While the Communication Station is running, data is continually being collected via existing connections, and system data is also collected for subsequent evaluation. This data is called up periodically by the CRM Server and can be displayed and analyzed there using monitoring tools. The gateway, which is installed on the Communication Station, is used to call up the collected data. Windows performance monitor
Monitoring Type
Communication Station Monitoring Activities / Functions CRM Transfer Service The SAP CRM Queued Transfer Service component logs the communication sessions between the mobile clients and the CRM Server in the TransferService.log file. The default file path is as follows: C:\WINNT\TransferService.log QmtCnfg.exe, a Release 3.0 tool, which can be used to view the current trace level and log file location Tracing the data transfer on the Communication Station is not required, only for troubleshooting.
TransferService.log
Daily
QmtCnfg.exe
2002 SAP AG
25
Communication Station Monitoring Activities / Functions DCOM Connector: Use the following menu path to access the DCOM Connector component monitor: Start > Programs > Middleware > DCOM Connector. Start the application and choose Monitor. The following tables explain the different areas of the monitor.
Monitoring Type
On demand
The following diagram shows the monitoring areas in respect to the Communication Station: The Communication Station (CommStation) is the link between the CRM Server and the mobile clients.
2002 SAP AG
26
Mobile Client(s)
Inbound Business Processing Queue Outbound Queue DB & Log
Comm. Station
R/3 System
Inbound Queue Outbound Queue SAP Business Processing
DCOM Connector
Log
Flow / Process Control Engine & other services R/3 Application Database CDB
2002 SAP AG
27
Mobile Client Monitoring Activities / Functions Queued Transfer Service The QmtCnfg program displays the connection status between the mobile client and the Communication Station. Client Console, metadata check, generation, comparison of BDoc structures Data bound from the CRM outbound queue to a specific mobile client is copied to the inbound queue of that mobile client. Data from the mobile client outbound queue is transferred to the inbound queue on the CRM Server. The inbound and outbound queues of the mobile client can be displayed using the Client Console. Operating system Windows NT Performance Monitor Database You can monitor the SQL-Server database on the Mobile client with the remote SQL-Server Monitor
Trace.log
Perfmon.exe
In case of an error message SAP Note 358507 SAP Note 433401 SAP Note 530317
2002 SAP AG
28
E-Mail Server
SAP Connect
SAPphone
Telephony Gateway
In ST03, Create Transaction Detail Profile for CIC0
In this section, CRM Middleware monitors specific for Interaction Center scenarios are described.
2002 SAP AG
29
Monitoring Type
Monitor SAPphone Interface Trace Files: switch the trace on or off and display the trace files. Alert Monitor: starting the collection method and view the alert monitor data (Option: Integration of the Alert Monitor into CCMS) SAPphone version: get the SAPphone version
SPHB Utilities > Trace > internal Trace Utilities > Alert Monitor > Start collection method or Display Utilities > SAPphone version ST03N
Upon error
Transaction Detail Profile for CIC0 monitor transaction CIC0 by using the transaction detail profile
2002 SAP AG
30
2.3 Verification
After adapting the recommendations described in this Best Practice, you can monitor your mySAP CRM solution using the transactions described in this section. Many CRM-specific transactions are not available in the R/3 back-end system. Those common to both the R/3 back-end system and the CRM Server are described together. Those handled or interpreted differently are explained separately.
2002 SAP AG
31
2002 SAP AG
32
The R/3 back-end system, the APO and the BW system should always be connected (except during the downtime for backup) The following table shows the different prefixes used for the different queues: Queue Name R3AI<Object name> R3AD<Object name> R3AR_<Object name> R3AU<Object name> CRM_SITE<Queue number> Systems R/3 CRM R/3 CRM R/3 CRM R/3 CRM CRM-> Mobile Client Description Initial. BAPIs destined for the R/3 back-end Delta Request Upload Synchronization BDoc types from the CRM Server going to the mobile client Outbound for simple replication of BDocs to receivers Queues to BW systems Queues to external systems
CSA_<Object name> CSA_MASS_<Object name> BW.. <Object name> EXT.. <Object name>
When mass updates are carried out (e.g. application data updates) in the R/3 back-end system, the data is shipped via the outbound queue to the CRM Server. This data is controlled via qRFC settings. Choose Information Version in transaction SMQ1 to determine the current qRFC version. For a detailed description of qRFC, refer to SAP Notes 193515 and 438015. Usage Menu path: Middleware Monitoring Queues Display Outbound RFC Queues Transaction: SMQ1 Handling Procedure Deleting a queue will lead to data inconsistencies between the sender and the receiver. E.g. deleting a queue that is in use between a mobile client and the CRM Server will lead to data inconsistencies between the CRM Server and the mobile client.
2002 SAP AG
33
Best Practice: mySAP CRM Field Sales Monitoring In this case, you must bring the client back to a consistent state. In some cases, you can do this by extracting the data from the CRM Server and sending it to the client. This restores the client state, as it is known on the server. Data that was not yet successfully transferred from the client to the CRM Server must be manually reconstructed. Local data that is not usually sent to the server may be also lost and you may need to reconstruct it manually. If you accidentally delete data from a queue that is not assigned to a mobile client, data may be lost and cannot be recovered (e.g. in APO, BW, R/3 System). In case of error, first try to solve the problem. If all other attempts to fix the problem fail, you may need to delete an entry in a queue. In this case, you must bring the different databases (R/3, CRM Server, CRM Mobile Client) manually into a consistent state. To do so, you must be familiar with the technical and business logic of the process involved. Problems and errors can occur during data exchange (such as program crash, lost connection between R/3 back-end and the CRM Server, and so on). Queues for the R/3 back-end should not wait too long. Possible causes for waiting can be e.g.: Error in the first BAPI in queue, Resource bottlenecks in the R/3 System, Logical sequence.
Check the status. If it is SYSFAIL or CPICERR, an error has occurred for the corresponding queue (most likely on the CRM Server). Double-click on the queue name and look at the status text of the first entry in the queue: you will find a short error message describing the error. After correcting the error, you can restart the queue. SAP Note 443900 contains useful information on how to analyze the error situations during the data exchange.
To display the function modules of the queue, double-click on the line. The following table describes the different information shown for a specific queue: Field Name Description
2002 SAP AG
34
Description Client Queue name Logical destination (specified in function call) Number of logical units of work (LUWs) Normal status READY: The first entry of the queue is ready for sending RUNNING: The first entry of the queue is just processing EXECUTED: The first entry of the queue is already processed, just waiting for confirmation of the receiver system NOSEND: no active sending, receiver has to fetch the records, deleting the record only after confirmation of the receiver
Error status SYSLOAD: No dialog work process free, the record will be automatically reprocessed by a background job SYSFAIL: error on receiver, look for a short dump there CPICERR: network or communication error, record will be automatically reprocessed by a background job Wait status STOP: Application locks the first entry of the queue temporarily WAITSTOP: Locked because of dependencies to other queues, because the other queue is just locked WAITING: locked because of dependency to another queues, there is the linked record not the first entry in the queue WAITUPDA: First record in the queue contains update function, for which the queue is just waiting NOSENDS: Queue is just waiting for debugging MODIFY: The data of the first LUW are just been modified Detailed information about the status can be found in note 378903. 1. Date 1. Time NxtDate NxtTim Wait for queue Oldest date of a queue entry Time of oldest queue entry Most recent date of queue entry Time of most recent date of queue entry Only used for linked queues, not in CRM
Handling Procedure
2002 SAP AG
35
Best Practice: mySAP CRM Field Sales Monitoring If you recognize many entries in one or more queues, take care of the following points: Only MSA/MSE: The corresponding client of the queue has not been connected to the CRM Server for a long time. You can check this in the column 1. Date. You should avoid situations when the mobile client has not synchronized his data for a long time. Many queues with a large amount of data can lead to a rapid growth of communication tables (e.g. ARFCSDATA) and have an impact on the system performance. The client should regularly connect to the CRM Server, in exceptional cases at least every two weeks. Only MSA/MSE: To reduce the amount of data the subscriptions and publications for a client should be as few as possible necessary from a business point of view. For all other queues not assigned to a client errors should be monitored hourly at least several times per day. Errors should be resolved and the queue restarted afterwards.
2002 SAP AG
36
It is strongly recommended to avoid deleting the queue or queue entries, because this can lead to data inconsistencies! The qRFC Inbound Queue Monitor, SMQ2, is not a critical monitor on the R/3 back-end system because there is normally no data to be displayed. Instead, monitor the outbound queue (SMQ1 or R3AM1) on the CRM Server. When the CRM Server tries to send data to the R/3 back-end and the system is not up, the data stays in the outbound queue of the CRM Server in status CPICERR. As soon as the data is in the R/3 back-end in-queue, it gets processed through to the respective application or to its appropriate end location. Data is not deposited in the R/3 back-ends in-queue and picked up later. It is processed immediately. The inbound queue information is stored in the following tables: Table Name TRFCQIN TRFCQSTATE TRFCQDATA ARFCRSTATE Description Status inbound Queue - Header of queue QRFC call condition Data of records Status table of the LUWs in the tRFC/qRFC target system (Header of the records)
Usage Menu path: Middleware > Monitoring > Queues > Display Inbound RFC Queues Transaction: SMQ2 1) Select the entry you want to view. 2) To view the details, press the Change View button (F8) and check for stopped or hanging queues.
2002 SAP AG
37
Descriptions Client Queue name Number of logical units of work (LUWs) Normal status READY: The first entry of the queue is ready for sending RUNNING: The first entry of the queue is just processing Error status SYSFAIL: error on receiver, look for a short dump there CPICERR: network or communication error, record will be automatically reprocessed by a background job ARETRY: Temporary error on receiver, record will be automatically reprocessed by a background job
Status Options
ANORETRY: Fatal error, processing by QRFC manager stopped, check together with application Wait status STOP: Application locks the first entry of the queue temporarily WAITSTOP: Locked because of dependencies to other queues, because the other queue is just locked WAITING: Locked because of dependency to another queues, there is the linked record not the first entry in the queue NOEXEC: Queue is waiting for debugging MODIFY: The data of the first LUW are just been modified Read SAP Note 378903 for further information related to the status.
Oldest date of a queue entry Time of oldest queue entry Most recent date of queue entry Time of most recent date of queue entry Destination of the sender Only used for linked queues, not in CRM
Handling Procedure An error has occurred, if during an object load with status Running (light yellow) the date, time and the number of blocks remain constant over the time or grows, if new entries are written to the end of the queue. No queue entries are processed here. General procedure: In the case of stopped queues, first search for the error causes and then fix them. Here are some possible error situations: If the status of the queue is SYSFAIL or CPICERR, an error is occurred.
2002 SAP AG
38
Best Practice: mySAP CRM Field Sales Monitoring By double clicking on the queue name and looking at the status text of the first entry in the queue, you will find a short error message describing the error reason. After the correction of the error, you can restart the queue. Check the short dumps (transaction ST22) on the receiving system, here the CRM system. An error in the queue data processing generates a short dump. Here you can find information about the cause and location of the error. Check further information about errors in the system log (transaction SM21) on the CRM Server. Have a look at the CRM Server Log.
The input queue on the CRM Server has status Ready, but the number of entries does not decrease:
Transactions cannot be performed because the system is overloaded. Check if enough system resources are available and check queue registration in SMQR. If the queue cannot be activated again, check the application log for possible application error messages. Data arrive from the R/3 back-end system but is still hanging in the inbound queue of the CRM Server: Check that the scheduler is set to active on the CRM Server (transaction SMQR) Follow-up problems can be found in the tRFC (transaction SM59) or in case of short dumps in transaction ST22.
If no delta data arrives from the R/3 back-end (no data in the inbound queue of the CRM Server), or the delta queue has the status STOP you can proceed as follows: At least one object has load status Running or Abort and has not finished. Wait until the object is finished or you have to force termination.
2002 SAP AG
39
Best Practice: mySAP CRM Field Sales Monitoring Transaction: SMQR Handling Procedure You can configure the Max_time parameter per queue by deregistering the queue and registering them again. Please do not set it too small (<60s), otherwise it causes a scheduling overhaed in your system. If the scheduler status is ACTIVE or WAITING, select Refresh (F9) at short intervals (approx. 1 second) and monitor the scheduler status as well as date and time of the field Last update (every 2 minutes). If the status of the scheduler is occasionally WAITING, and the date, time in field Last update (every 2 minutes) changes every few seconds without a decreasing number of queue entries in SMQ2 this indicates that the qRFC scheduler does not get enough resources. The qRFC scheduler may then not start any asynchronous RFC calls (all dialog work processes are blocked) with the result that the qRFC inbound queues are not processed. You can check whether the system can provide enough resources by using transaction SM50. A few dialog work processes should remain in status waiting. Otherwise the system is overloaded. Important: perform the setup depending on the number of workprocesses as well as CPU resources. If you need to specify a special RFC server group for sending please maintain it in transaction RZ12 and specify the RFC server group name in transaction SMQS>edit>change AS-group.
2002 SAP AG
40
Usage Menu Path: Middleware Monitoring Message Flow Transaction: SMWMFLOW Activation To activate the statistics, choose the transaction SMWMFLOW and execute Goto Activate statistics. If you want to check whether the application statistics from the R/3 kernel have been activated, choose Kernel application statistics. The checkbox for Middleware Message Hub Statistic must be active. So that the statistics are actually activated, you must have first scheduled the background job SAP_COLLECTOR_FOR_PERFMONITOR. Choose Middleware Message Flow Statistics. You can toggle the creation of statistics on and off with On/Off. When On is selected, the background job RSMWM_BSTAT_COLLECTOR is automatically scheduled (status: Collector: On).
Workload from database Choose WorkloadWorkload from database. Select one or all (TOTAL) R/3 instance(s) and specify the required time period. This statistics records the progress of BDoc messages in the CRM Server (including the retention period in inbound queues). The retention period in the outbound queues is not recorded. The single records are directly written to the message flow before the completion of processing. In contrast to the application statistics, these statistics also contain data from the message header of the BDoc messages concerned. The report RSMWM_CHECK_STATISTIC_RECORDS displays single records of the application statistic. It displays related statistical data records together (i.e. a single BDoc type with its services). You can also view the entire statistical data record here by double clicking. Just as in case of transaction ST03N, the single records created in this way are available at least for the current day or the last 2-3 days, depending on settings in ST03N.and can also be displayed. These records are automatically aggregated by the CCMS data collector to form the following statistics (the sum of the total and detailed response times, as well as the average total and detailed response times): By BDoc type (including the times for the services)Per service (for all BDoc types)Per BDoc type and service. Just as in the case of ST03N these statistics are available for days, weeks and months. Column Records displays the numbers of flow runs of the different BDoc types.
Alternatively, in SMWMFLOW, choose Workload from database Per service Here you can get the workload for individual services per BDoc type. You can also display the workload per BDoc type (Detail).
2002 SAP AG
41
Best Practice: mySAP CRM Field Sales Monitoring This overview does not include the processing time for the following: Inbound adapter R/3 inbound adapter Flow itself
Therefore, the sum of the total response times of all services can be significantly smaller than the total response time of the BDoc type you have chosen before. Last minutes workload Choose Workload last minutes workload (Set time interval: Analyze last 15 min for this Instance) Here you can get statistics about the current workload of the message flow in your system (for BDoc types and for CRM services). To display statistics for the Middleware services, you can also choose Goto Service statistics. In the service statistics you can see Detail statistics for individual services as well as the workload for individual services per BDoc type. In all detail statistics, to display individual statistics for the selected BDoc type or service, choose Single steps Alternatively, in SMWMFLOW, choose Last minutes workload Per service Message-flow statistics Press the Message flow statistics button. The Data Collector starts automatically every hour to provide the required data. If you need the latest data for your analysis, you can manually start the data collector at any time by choosing Run Collector. The statistics relate to one week and to different instances. The header data shows the display and your selection criteria. The results show the single records that the statistics refer to. For each record the BDoc type, client, site, queue and name of the CRM Server are displayed, as well as the retention time of the corresponding BDoc type on the CRM Server.
2002 SAP AG
42
Best Practice: mySAP CRM Field Sales Monitoring Search for SAP Notes that solve the problem. If there is no solution, create a customer message.
Note: Use SMO8FD after a new BDoc definition has been transported.
2002 SAP AG
43
If the status symbol for an object is RED, refer to SAP Note 429423 for initial load problem analysis.
Usage Menu path: Middleware Monitoring Queues Monitor R&R Queues Transaction SMOHQUEUE Field Names Name AC_EXTRACT Descriptions Name of queue: Represents those extract jobs triggered directly from the Administration Console, for example for a data reload to a mobile client to recover the local SQL database Contains extract jobs broken down by BDoc types of the type bulk. Contains extract jobs broken down by BDoc types (the jobs are created by realignment) Contains realignment jobs for individual BDoc types whose responsibilities must be checked Contains subscription checker jobs (these are generated in the Administration Console, if a subscription is changed) Current number of entries (messages) in the queue Current number of active tasks
2002 SAP AG
44
Best Practice: mySAP CRM Field Sales Monitoring Release queues for processing by setting their status to Released (yellow light) Reset released queues to Hold (red light) Interrupt queue processing (Status Running green light) by setting the status to Hold.
Handling procedure Normal condition: all lights green or yellow, number of queue entries is decreasing, if no new queue entries are created. If an error occurs during processing the Queue demon automatically sets the queue to hold. An error symbol appears in the last column. You can select it to display the error message. When you release the queue again, the error status is reset. If you release a queue after the Queue demon has started, the status of the queue changes to Running, provided it contains entries for processing. If all entries have been processed (number of entries is zero), the status changes to Released again.
If the number of entries in a queue remains constant over a period of time, it may be necessary to manually trigger the processing jobs by clicking the green flag corresponding to the queue. If a queue contains entries, you can display details of the individual entries (such as the number of sites affected) by clicking on the number of entries. You can delete individual entries by selecting the checkbox in the first column of the record and choosing Delete. To update the information choose Refresh, but: Try to avoid deleting entries; this leads to inconsistencies of the mobile client local database. Refer to note 429627 for implementing the authorization checks necessary to avoid this kind of situations.
2002 SAP AG
45
Registered
This column is displayed only in the case of inbound queues. It indicates, if a queue has been registered (R), has never been registered (empty) or has been deregistered again (U). A red symbol indicates that there are entries in the respective queue, that the queue is however not registered or that there are problems with the Queue Scheduler (for example status SYSFAIL or CPICFAIL). A yellow symbol refers to the same problems, but indicates that there are no entries in the queue. A green symbol indicates that there are no problems.
First Date / First Time Last Date / Last Time Site description
Informs you when the oldest entry was placed in the queue. Informs you of when an entry was last placed in this queue. Describes the respective site and was defined when this site was created in the Administration Console.
Usage Menu path: Middleware Monitoring Queues Display Mobile Site Queue Information Transaction SMWMQUEUES Handling Procedure Shows the last time of connection. Here you can see whether a client did connect for a longer time (also shown in transaction SMQ1), Assignment of site name and queue name (also shown in transaction SMOEACSMOEAC).
2002 SAP AG
46
Best Practice: mySAP CRM Field Sales Monitoring You can monitor the Conntrans performance per site, queue or Commstation. Otherwise you can check the performance per time range, for instance, day or week. The statistic you get shows upload and download times and transferred mount of data. It is also possible to get information about failed session between Commstation and CRM server.
The Mobile Client Message Recovery displays the status of BDoc messages sent from the CRM Server to the Mobile Clients (inbound processing) and allows their reprocessing: Usage Menu path: Middleware Monitoring Mobile Client >Message Recovery Handling Procedure Processed messages have successfully been processed from the inbound queue into the database of the Mobile Client. Unprocessed messages could not be saved on the client database. These BDoc messages are deleted by the system, a reporting message is sent to the CRM Server. The corresponding BDoc messages can be sent again to the Mobile Client by calling the transaction Message Recovery. The complete information on the BDoc message status on the Mobile Client can be accessed via the Client Console.
2002 SAP AG
47
Best Practice: mySAP CRM Field Sales Monitoring Shows the open connections of the DCOM Connector Column Headings Destination State Initial: created but not yet allocated Connecting: connecting to server Active: allocated by an object Pool: resource was returned to pool without cleanup Cleaning: R/3 context is being released Clean: resource was returned to pool in a clean state Closed: resource is closed and not available Sys Host ID SAP User Rem. Vers. Rem. Codep. Proc. hRfc Loc. Vers. Loc. Codep. Conv. ID T Calls Last Function Name of application server Hostname of application server System number (port number is 33## and 32##) User ID in R/3 Version on application server Code page on application server. Process ID Handle to RFC connection Version of DCOM Connector Code page of DCOM Connector Conversation ID like in SAP gateway monitor Trace Number of calls since startup Name of last function being called Descriptions
2002 SAP AG
48
Best Practice: mySAP CRM Field Sales Monitoring Object ID (session ID) Site name Queue name Name of the PC from which the session was opened Name of the Windows user Session status
Session performance data Number of transactions sent to the R/3 System Number of table entries sent to R/3 Number of transactions received from R/3 Number of table entries received from R/3 Compression rate for data received from client Compression rate for data sent to client Processing time on the DCOM Server for data received from R/3 and sent to the client Processing time on the DCOM Server for data received from the client and sent to R/3 Session status Detailed information for the functions Read, ReadConfirm, Send and SendConfirm: Number of calls to R/3 R/3 response time Network time between Mobile Client and DCOM Server Response time of the client database Transmission rate for communication of DCOM Server with R/3 Transmission rate for RFC communication between client and DCOM server Access rate in the client database (bytes per second)
When analyzing network conditions, PERFMON should be run locally on the Communication Station rather than from a remote server. When running PERFMON from a remote server,
2002 SAP AG
49
Best Practice: mySAP CRM Field Sales Monitoring network statistics can be affected due to the network load caused by PERFMON itself. Other network-based applications, such as PCAnywhere, also create a significant network load that may invalidate any network statistics measured while connected. Usage Verify that the Performance Monitor is installed. Set up the counters and the log file (modify/set the log file and chart settings) Ensure that no other services or programs are running that may impact the measurement (such as programs causing network or CPU load). Perform the measurement Extract the relevant counters (export them to a file) Calculate the relevant quantities Interpret the results
For further details, go to www.microsoft.com Support Knowledgebase and see the White Paper Measuring performance-relevant data using PERFMON on Windows NT. In Windows NT/2000 start PERFMON by choosing Start Run and enter PERFMON. Handling Procedures Set up PERFMON as follows. 1. Measure the CPU utilization on the server or PC. From the PERFMON menu, choose Edit Add to Chart / Add Item Select in PERFMON: Object, Processor, Counter, % Processor Time 2. Determine the size of dataset transferred for analysis. Select in PERFMON: Object, Network Interface, Counter, Bytes Received/sec Note: Depending on the network connection, the name of the category and item of the abovementioned examples may differ. When selecting the network interface counter it is possible that various instances (= lines) are offered. If you are not sure on which network line the traffic takes place, choose all instances. You may have to start processes at the NT level monitoring network traffic. 3. To start your analysis, monitor during the peak time when Mobile Clients are uploading to the CRM Server. To save the performance data and transfer it into a spreadsheet program, choose File Export Chart. 4. Evaluate your analysis.
SAP connect provides a standard interface for external communication, which supports sending using telecommunication services, such as faxing, pagers (SMS), Internet and X.400, as well as sending to printers and between several SAP Systems. It enables external communication components to be connected to the SAP System.
2002 SAP AG
50
Best Practice: mySAP CRM Field Sales Monitoring As of Release 6.10, SAP connect also provides a direct connection to the Internet using the SMTP plug-in of the Web application server. This enables you to send messages directly to Internet, fax, and paging addresses without using an additional external communication system. Transaction SCOT can be used for SAP connect administration.
Handling Procedures
SAP connect administration provides various views of your communications environment. Each view shows the environment from a certain viewpoint: Nodes / System Status: You see all nodes defined for your System and their status when you choose View > System status or View > Node. Choose the DISPLAY Button to view the node data. Jobs: You see all Jobs defined for sending mails of faxes from your System. The list shows for each job the time for the next run and the last runs that transmitted mails or faxes. When you mark one entry and chose the DISPLAY button you jump into the job details where you can find the job logs. Routing: The view of routing (View > Routing) shows, for each communication method, which address areas are processed by which node. The display also tells you if a communication method is not available in your system or is processed using another communication type. Alert Monitor: Choose Utilities > Alert monitor > Start data collection method to start the data collection for the monitor and then Utilities > Alert monitor > Display to see the results of this collection. (Option: Integration of the Alert Monitor into CCMS) Send orders: to get an overview of the send orders choose Utilities > overview of send orders. You can specify a time period, the send status or the sender. Descriptions Status of send process (i.e. transmitted, waiting, send, errors,) Transmission Method (i.e. via Telefax, via Remote SAP, via Internet,) Document Title Sender Recipient Send date Send time Message: Link to further details like status text, long text and transmission history
Field Names Status Trans. Method Doc. Title Sender Recipient Send date Send time Msg.
(Double-click on the entries to see more details.) For error analysis, you can use the trace functionality by choosing Utilities > Trace > Internal Trace. Here you can set the trace for inbound or outbound direction and display results.
SAPphone integrates telephony functions in SAP applications. This allows data to be exchanged between computer and telephone processes.
2002 SAP AG
51
Best Practice: mySAP CRM Field Sales Monitoring Third-party manufactures can program a direct connection to the RFC interface or use the SAPphone server delivered with the SAP system. The SAPphone server serves as a nexus between the SAPphone RFC interface and the TAPI standard interface that Microsoft has defined for telephone integration solutions. If the SAPphone server is used, manufacturers of TAPIcompatible telephony products do not have to program an individual connection to the SAPphone RFC interface. The functions for starting and monitoring the telephony environment are available in transaction SPHB. The existing work centers are displayed there for each telephony server in the navigation tree. You can also display other information about each telephony server. In the standard view, the telephony servers are displayed with name and description, the work centers with extension and user name of the work center user.
Handling Procedures
To enable communication the SAP System and the SAPphone server (or the self-programmed gateway software) you need a configured RFC connection between them. The following settings are required: For outbound calls, an RFC destination should be maintained in transaction SM59 For inbound calls, an RFC destination should be maintained in the Saprfc.ini or the gateway software
You can see the name of RFC connection for the corresponding telephony server in transaction SPHB in tab strip Server attributes. By double-clicking on the destination name you can jump into the destination settings in SM59 and perform e.g. connection test or maintain required settings. Trace Files: Choose Utilities > Trace > internal Trace. Here you can switch the trace on or off and display the trace files. Alert Monitor: Choose Utilities > Alert Monitor > Display or start collection method. After starting the collection method you can view the alert monitor. (Option: Integration of the Alert Monitor into CCMS) SAPphone version: Choose Utilities > SAPphone version to get the SAPphone version
Use this transaction to monitor and administrate the ICM (Internet Communication Manager). The ICM receives and sends requests (i.e., in a server role, incoming HTTP requests) from or to the internet. The SAP Web Application Server can serve both as a (WEB) server and as a client. As a server, ICM accepts incoming HTTP requests and forwards them to the Internet Communication Framework for processing. As a client, requests (i.e. SMTP e-mails) are sent from the SAP server via ICM to any Internet server (i.e. Mail server).
Handling Procedures
Status: you can check the status of Internet Communication Manager by calling transaction SMICM. The ICM status should be "runs" with green lights. Services: Choose Goto > Services to obtain the service list (e.g. SMTP, HTTP(S)) with corresponding ports. The column Active shows you whether the service is running. Trace Files: Choose Goto > Trace file or Trace level. Here you can display or reset the trace file and set the trace level (values between 0 and 3 are permitted, default is 1). For normal operation, 2002 SAP AG 52
Best Practice: mySAP CRM Field Sales Monitoring the ICM trace level should be set to 1. Higher trace levels should only temporarily be used during active error monitoring.
Analyzing the gateway trace gives you information about the RFC connection between the CRM Server and the Communication Station.
Handling Procedures
Check if the connection is broken or still existing and check for CPIC errors.
Further you can use the Analysis Views to display: Transactions used and the users that call them
2002 SAP AG
53
Best Practice: mySAP CRM Field Sales Monitoring The time profile The ranking list The RFC profiles
General Threshold Values - ST03 Name Dispatcher wait time Database time Processing time Avg. response time Handling procedure As part of the SAP Basis, the Workload Monitor does not display any CRM specific monitoring data. However the RFC response time statistics should be monitored relative to the RFC communications between the R/3 back-end system and the CRM Server. In general, it should be analyzed using the same procedures as those used in a non-CRM system environment. Monitor online transactions by using called transaction detail profiles If you are implementing Interaction Center and/or CRM Online Scenarios, you have to check the response times for transaction CIC0 or for the corresponding CRM Online transactions. In order to be able to monitor these transactions by using the so-called transaction detail profiles we recommend to configure them in the transaction ST03(N). Recommendation: 1. Call transaction ST03(N) 2. Change the user mode to 'Expert mode' 3. In upper launch pad choose 'Collector and Performance DB' -> 'Workload Collector' -> 'Statistics to be Created' 4. In section 'Create transaction detail profiles for' enter the transactions you would like to have analyzed (e.g. CIC0). 5. Click button 'Save values' Afterwards, you can display and analyze collected data in Transaction Profiles -> Details by double-clicking on the corresponding transaction name. Analyzing RFC Profiles In all CRM Scenarios, large part of processing in CRM is done by RFC. The RFC statistics should be therefore analyzed in detail. Choose: RFC Profiles, you can check client (sender) and server (receiver) records in detail. Keep in mind, that QRFCs and TRFCs are processed as function ARFC_DEST_SHIP. All other RFCs are displayed by function module in this statistics. You can double click to get details of the call type. Select the function module and look into SE37 to get the short description. Check to see which functions are time consuming or very often executed or transport high mount of data. SAP Threshold values <= 50 ms < 40% of (response time - dispatcher time) < 2 x CPU time < 1000 ms (for records without RFC-subrecords)
2002 SAP AG
54
Best Practice: mySAP CRM Field Sales Monitoring The Database Performance Monitor does not display any CRM specific monitoring data. It should be analyzed using the same procedures as those used in a non-CRM system environment. Exception: Check the indices on the tRFC and aRFC tables (ARFC* and TRFC*). For MSA/MSE: monitor the disk I/O in high-load phases to find possible bottlenecks. See also transaction DB02. Refer to R/3 Monitoring Best Practice.
2002 SAP AG
55
Best Practice: mySAP CRM Field Sales Monitoring By monitoring work processes in the SAP System (transaction SM50) or from outside the system (program DPMON at operating system level), you can determine the status of the work processes in relation to the PRIV mode. If work processes are often switched to the PRIV mode, you must increase the extended memory and/or adjust the limit for the extended memory). In this case, you must consult with SAP. You can use transaction SM66 to evaluate this information for all the work processes. You can determine the swap space requirements at R/3 and operating system level. Many monitoring tools are operating system dependent.
Transaction SM66 shows the SAP System work process overview for the local application server and for all other SAP application servers that are connected to the same SAP database (such as all application servers connected to the database and central SAP instance). Usage With help of this transaction you can detect long running transactions. Monitor periodically, and also check if there are sudden performance problems. Handling Procedures SM66 check for ARFC programs running Make sure that there are enough work processes free for online users, if all processes are blocked, check the RFC parameters. Are there work-processes that are stopped or locked by CPIC to determine whether there are enough work processes configured for the system? In SM50, ARFC jobs show up as entries for the background work processes of rescheduled tRFCs..
Check whether it says no subrecords Check RFC subrecords to see who has done what.
2002 SAP AG
56
Delete a single message, with TID 0A1089630DEC3D491F782EA9 The following system log entry is created 15:32:09 DIA 3 800 D036792 SMQ1 Q23 Missng:TSL1TE,Q23):D036792&0A1089630DEC3D491F782EA9 Error in SM21: Q23, user deleting and from which transaction as well, which client If something is deleted in system log
Technical details File 170136 Position 0000329040 Entry type Message ID Q2 3 Variable parts D036792&0A1089630DEC3D491F782EA9&CRM_SITE_000000000000100,CRM_SI
2002 SAP AG
57
Best Practice: mySAP CRM Field Sales Monitoring Example: Deletion of whole queue (here from the inbound queue) (Q22, Q25)
If a whole queue is deleted, this information is displayed (the TID are not shown anymore): 15:40:14 DIA 3 800 D036792 Q22 Missng:TSL1TE,Q22):D036792&CRM_SITE_000000000000100 Example: Deletion of a single message from the R&R Queues (CM1)
Client information is missing on this screen, but it is in the process. Example: Release and hold R&R Queues (CM0)
2002 SAP AG
58
2.4 Troubleshooting
This chapter describes problems that can arise in the CRM environment and helps handling and solving these problems. The possible problem solving strategies are described either as flow diagrams or as problem descriptions.
Whatever the reason, the exact cause of the problem must be determined and the problem fixed before the data transfer can be properly completed. The download or upload of objects should only be restarted after the errors have been found and repaired. The following problem analysis diagrams should be used as guides in correcting errors occurring during the initial download, during delta downloads and in troubleshooting upload problems. The basic recommendation is to first ensure that the download works properly before trying to correct any upload problems. Middleware specific situations are covered in the following flow diagrams - the CRM application settings may need to be handled by the application personnel. 2.4.1.1 Error Detection in Delta Download
If you experience problems with delta load of some objects, you can refer to the flow diagram below. This is a logical sequence of checks to analyze the problems and helps to determine and to eliminate the causes of the errors. For further information, refer to: SAP Note 430980 and/or to SAP Library CD Error Detection in Delta Download.
2002 SAP AG
59
NO
OLTP System?
YES
Event Control
NO
Active?
Table TBE11 Parameter: BC_MID, ACTIVE: X Parameter: NDI, ACTIVE: X
Table CRMCONSUM CONSUMER: CRM (or consumer application, e.g. BBP) ACTIVE: X TEXT: CRM (or a descriptive text) Table CRMRFCPAR CONSUMER: CRM (see table CRMCONSUM) OBJNAME: * RFCDEST: <RFC Dest. of CRM System> DOWNLOAD: * INACTIVE: <blank> DISCARDDAT: <blank> USE_IN_Q: X, SEND_XML: X HOLD_DATA: <blank>
2002 SAP AG
60
Best Practice: mySAP CRM Field Sales Monitoring (Continued from the previous page)
OLTP
CRM
YES?
CRM System ST22 OLTP System ST22 Analyze and fix problems
YES?
NO?
YES?
NO?
2002 SAP AG
61
Best Practice: mySAP CRM Field Sales Monitoring (Continued from the previous page)
NO?
OLTP SMQ1
YES? CRM SMQ2 Restart the queues manually CRM SMW01 Release queue (manually unblock queue)
NO
CRM - R3AM1
YES Ensure that all objects have GREEN status OLTP SMQ1 Ensure that all initial download objects are completed
NO
YES
CRM - R3AC4
Activate the missing entries manually by using change mode and then save
2002 SAP AG
62
Best Practice: mySAP CRM Field Sales Monitoring (Continued from the previous page)
NO
YES
CRM SMQR
All R3AD* queues must have R in Type column
YES
NO
Do FILTER criteria exist for objects that have not been downloaded?
YES
CRM R3AC1
NO Delete unwanted filters that may be preventing an object to download from OLTP Check with the application people before deleting any filters!
2002 SAP AG
63
Best Practice: mySAP CRM Field Sales Monitoring (Continued from the previous page)
Does the CRM Inbound queue stop after processing each object?
YES
CRM SMQ2
Check for errors NO CRM Production client table SMOFPARSFA Check for BREAK-POINTS (parameter BREAK_POINT) Delete the BREAK POINTS or deactivate them by entering a non-existent user-name for each
NO
YES
SMO8FD Check whether the Message Flow has been generated. GNRWB Generate the Message Flow for all objects
YES
DONE
2002 SAP AG
64
2.4.1.2
Error Detection in Initial Download SAP Notes 429423 and 350176 SAP Library CD Error Detection and Correction
References:
Tablespace problems?
YES
NO
CRM SM21 the system log shows the affected tablespaces Extent tablespace
Are the newest Support Packages installed - R/3 Plug-In? OLTP System? CRM System?
NO Check SAPNet for newest Support Package releases HTTP://Service.SAP.com/swcenter-main Upgrade to newest versions
YES
NO Table CRMCONSUM CONSUMER: CRM (or consumer application, e.g. BBP)) ACTIVE: X TEXT: CRM (or a descriptive text) Table CRMRFCPAR CONSUMER: CRM (see table CRMCONSUM) OBJNAME: * RFCDEST: <RFC Dest. of CRM System> DOWNLOAD: * INACTIVE: <blank> DISCARDDAT: <blank> USE_IN_Q: X, SEND_XML: X HOLD_DATA: <blank>
65
Best Practice: mySAP CRM Field Sales Monitoring (Continued from the previous page)
NO?
YES?
Test RFC connection between the systems OLTP System CRM System CRM System OLTP System CRM - SM58 check for errors in tRFC
YES
NO
R/3 - SMQ1 check for errors, restart queue in case of capacity overload CRM- ST22 check for short dumps
YES
NO
YES CRM SMQ2 check for errors, often there are customizing errors CRM - SMW01 analyze errors, correct the customizing settings CRM SMQ2 Restart the processing of the queue Check note 309734
NO
CRM SMQ2 restart the queue CRM - SMQR check registration of queue
Best Practice: mySAP CRM Field Sales Monitoring (Continued from the previous page)
YES
DONE
Furthermore, there are several other problems related to the initial download: - R3AM1: the Download Manager shows that some objects remain in status WAIT: The reason is that this object has a parent object, and the initial load of the parent object was not yet completed successfully. Whether or an object has a parent object can be seen in transaction R3AM1 by the "HasPar" indicator. When it is set to "X", then the respective object has parent objects. These parent objects can be seen in table SMOFOBJPAR. - R3AM1: the Download Manager shows only few Objects in status RUNNING: The maximum allowed number of loads and requests is already being processed. This number can be set in table SMOFPARSFA via the MAX_PARALLEL_PROCESSES parameter (default = 5). If the number of running loads and the number of running requests in total is as much as or higher than this number, the remaining loads have to wait until a process is free again. - The R3AM1 transaction shows that an object from the initial download has a status of RED The initial load has not finished successfully. Analysis Errors occurred in the CRM System during the initial download and as a result the data was not posted into the CRM database. In order to be able to restart the data from the queue after elimination of the error an exception was probably triggered. Normally, the problem is caused by missing/incorrect customizing of the CRM online application. Handling: See SAP notes 429423, 350176 The reason why the initial load was not successful must be determined. Error messages and traces should be examined to see where the problem(s) might be. Use transactions SMW01 (BDoc Viewer) or SM08FT (Flow Trace) to view the error messages Correct the error based on the error message text Use transaction SMQ2 to restart the queue Use a similar procedure for subsequent errors
2002 SAP AG
67
Best Practice: mySAP CRM Field Sales Monitoring Do not a complete restart of the initial download. The download will resume from the point where it stopped after the error has been corrected. Customizing of the CRM online application as specified in the SAP CRM Setup and Load Guide may have to be done to avoid subsequent errors.
2.4.1.3
Error Detection in Upload SAP Note 431345 SAP Library CD Error Detection in Upload
References:
2002 SAP AG
68
NO Does the Delta Download R/3 change i.e. the address of a business partner Analyze errors using Error Detection in Delta Download
YES
YES
NO
CRM SMW01 analyze all entries marked red or yellow find out which steps the message has already run through CRM - SMQ1, SMQ2 check for dependant queue entries R/3 SMQ1, SMQ2 check for dependant queue entries CRM + R/3 ST22 check for short dumps CRM SMQ1 Debug LUW (Note 337753)
2002 SAP AG
69
Best Practice: mySAP CRM Field Sales Monitoring (Continued from the previous page)
NO
YES
CRM SMW01 analyze relevant entries CRM SMQ1 analyze similar message by using Debugging (Note 337753)
YES
NO
DONE
2002 SAP AG
70
NO?
YES?
Check the profile parameters RZ10: icm/plugin_x: PROT=SMTP,PLG=/usr/sap/<SID>/SYS/exe/run/smt pplugin.dll Icm/server_port_x: PROT=SMTP,PORT=8025 (default Port) Check path for the Plugin (AL11)
NO?
Check the settings for the SMTP node SCOT: YES? Mail host and Mail port correct Address area correct Node active
NO?
2002 SAP AG
71
Best Practice: mySAP CRM Field Sales Monitoring (Continued from the previous page)
YES?
Analyze errors -SCOT: NO? Alert Monitor, send Orders (see above)
YES?
External Software: Look for error messages on the mail server) NO? Call SAP Support
DONE
2.4.3 CRM Server Performance problems or high I/O load due to excessive traces and logs being written
The performance of the CRM server is poor and may have some of the following symptoms Slow response times High I/O load Disk space increases relatively fast Great data quantities are written to the database logs Database is in a standstill condition The Middleware Log and Flow control Trace tables are larger than 100 MB (transaction DB02 can be used to check the size of the tables)
Affected tables:
2002 SAP AG
72
Best Practice: mySAP CRM Field Sales Monitoring SMW3_BDOC, SMW3_BDOC1, SMW3_BDOC2, SMW3_BDOC4, SMW3_BDOC5, SMW3_BDOC6, SMW3_BDOC7 (Tables for Administration of BDoc messages) SMWT_TRC (Table for Middleware Trace). Analysis Possible causes High trace level settings are activated on the CRM server The log level may be set incorrectly
Handling Check the relative sizes of the above listed tables using transaction DB02 Detailed Analysis Tables and Indexes Check the following traces, optimize their settings, and schedule a periodic reorganization of the tables as described in note 206439: As of Release 3.0: Set default trace level '1' for all trace environments except for 'G' (generation) of the Middleware trace. For this purpose, choose 'Middleware' -> 'Administration' -> 'Define Middleware Parameters' (R3AC6) from the SAP Menu and check entries with the following structure in the displayed table (SMOFPARSFA): Column Key ParamName ParamName2 Param.Wert Value CRMGENERAL TRACE-LEVEL ENV=m LEVEL=n
Where m is the trace environment and n the trace level that can be a number between 1 and 4. If this is higher than 1 in certain entries, except for those where the 'ParamName2' column contains the string 'ENV=G' (trace environment generation), set the default value 1 and then save the changes. The default value of the trace level for the trace environment generation (the entry where the ParamName2' column contains the string 'ENV=G') is 4. Scheduling the reorganization Reference SAP Note 206439 Create a background job MW_REORG using variant SAP_MW_REORG and schedule the SMO6_REORG program as periodic job for execution every night. When you use the variant SAP_MW_REORG all trace and log data that is older than a week will be deleted. If you want to keep the data for a longer or shorter period create a variant for the SMO6_REORG report via Transaction SE38 and use it instead of SAP_MW_REORG for the job scheduling.
2002 SAP AG
73
2.4.4 Mass changes of data (creating, changing) on the OLTP system leads to reduced system performance
Due to all of the dialog and background work processes becoming occupied the OLTP system has poor performance. Analysis: The outbound queue (SMQ1) of the OLTP system shows that there are between several hundred and several thousand different queues waiting to be processed. A large amount of the changed data from the OLTP system is intended for the delta download to the CRM Middleware Server and is therefore stored in the outbound queue of the OLTP before being transferred to the middleware server. For performance reasons qRFC tries to transfer as many queues as possible, as fast as possible. The result is that many parallel queues (with different queue names) are generated. The total number of queues that can be generated for parallel processing is limited by the possible names that can be assigned to the queues. Handling See SAP Note: 356228.
2.4.6 System performance degrades as the size of the tRFC/qRFC tables increases
There is a high system interface load because 1 or more of the tRFC/qRFC tables have become too large. Affected tables:
2002 SAP AG
74
Best Practice: mySAP CRM Field Sales Monitoring ARFCSSTATE ARFCRSTATE TRFCQIN TRFCQSTATE Analysis 1 or more of the tRFC and qRFC tables have become bigger than 500 MB. The large tRFC and qRFC tables affect the performance of the tRFC and the qRFC interfaces. Handling Use transaction DB02 Detail Analysis to check the relative sizes of the tables. Reorganize these tables regularly to avoid future performance problems. Reorganization will decrease the sizes of the tables. Refer to the SAP Note 375566 for details concerning administration of these tables Note: Never delete the contents of this tables otherwise data inconsistencies might occur. ARFCSDATA TRFCQOUT TRFCQDATA
2002 SAP AG
75
Checklists
SAP Basis (CRM Server and R/3 Back-end Server) SMGW ST02 ST03N ST04 DB02 ST06 SM50 SM66 ST22 SM21 SM13 STAD
CRM Server SMWP SMQ1 SMQS SMQ2 SMQR SM58 SMWMFLOW SMW01/SMW02 SMWT SMW03 SMO8FD R3AM1 R3AR3 SMWMQUEUES SMOHQUEUE SMWMCOMM CMWQ
2002 SAP AG
76
2002 SAP AG
77