How To Analyze and Troubleshoot The Cancelable RPC Dialog Box

How to analyze and
troubleshoot the Cancelable

RPC dialog box
Nagesh Mahadev and Hamza Hassen

Support engineers
Enterprise Business Applications
Microsoft Corporation
Nagesh Mahadev
Nagesh Mahadev joined Microsoft as a Support
Engineer on the Microsoft Exchange Client/Server
Infrastructure team in June 2001. He is currently a
Support Engineer in the Exchange Administration
team. He has worked on site together with some of
Microsoft’s largest Premier customers to resolve
mission critical and highly complex issues that
involve their messaging infrastructures. Nagesh has a
Master of Science degree in Information Systems
from Arizona State University.
2
Hamza Hassen
Hamza Hassen joined Microsoft as a support engineer
on the Exchange Admin team in April 2001. Before
joining Microsoft, Hamza held different positions.
These include instructor, support engineer,
consultant, and junior programmer. Hamza has a
bachelor’s degree in physics from Addis Ababa
University in Ethiopia.
3
Objectives
This Support WebCast discusses how to collect
and analyze data to troubleshoot the Cancelable
RPC dialog box. The presenters will discuss
various reasons that the Microsoft Outlook
"retrieving data" message may appear and how
to use the Microsoft Exchange Best Practices
Analyzer (ExBPA), the Performance Monitor
Wizard, Network Monitor traces, and other tools
to troubleshoot this issue.
4
Agenda
The basics of MAPI connectivity to Microsoft
Exchange Server
History of the remote procedure call (RPC) dialog
box
Common causes
Tools to use to collect data
How to analyze the data
5
How MAPI clients access Active Directory®
Microsoft Outlook® 2000 SR-1 and earlier versions

All requests to the directory are
sent to Exchange 2000 or
Exchange 2003 servers.
The Exchange server contacts
the global catalog on behalf of
the clients.
Clients Exchange Global
server catalog
MAPI clients running Outlook
2000 Service Release 1 (SR-1)
and earlier versions use the proxy
functionality.
Earlier versions include Outlook
2000, Outlook 98, Outlook 97,
and the Exchange Server 5.0
client. Outlook 2000
6
Accessing directory information
Later versions of Outlook
All requests to the

directory are sent to
global catalog
servers.
Clients Exchange Global
server catalog
Later versions of
Outlook such as
Outlook 2000 SP2 and
later service packs,
Outlook 2002, and
Outlook 2003
7
Outlook 2000 SP2 and later versions
Opening the first session to a global catalog
Exchange
server Global
Reference of the
catalog
Logon: contact name of one of
Exchange server the global
catalogs Uses the global catalog for all
queries to the directory during
that session
Stores the name of

the global catalog in
Outlook 2002 the user's profile
8
Outlook 2000 SP2 and later versions (2)
Successful connection to the GC
Available server
Global
Uses GC1 Catalog 1
Logon: contacts GC1

which is stored in the
registry
Outlook 2002
9
Outlook 2000 SP2 and later versions (3)
Failed connection to the GC
Server not
available
Exchange
server Global
Requests Catalog 1
another global
catalog
Response:
GC2
Logon: contacts GC1 which is
stored in the registry
Stores global
catalog name
in the user's Uses GC2 for all queries to the
Outlook 2002 Global
profile directory during that session
Catalog 2 10
Finding the global catalogs
Site 1
Site 2
GC 2
GC 3
GC 1
Exchange
server
Outlook 2002 11
History of RPC dialog box
Introduced in Outlook 2002
Not intended as an error message, but to
improve the user experience
Generally indicates that the client is taking an
unusually long time (more than 5 seconds) to
contact or to receive information from the
Exchange server or global catalog server
Outlook 2002 message
Outlook is retrieving data from the Microsoft Exchange
Server server_name.
Outlook 2003 message
Outlook is trying to retrieve data from the Microsoft
Exchange Server <servername>.
12
How do we see them?
Most cases take one of these forms
RPC messages (Outlook 2003 and Outlook
2002)
Client stops responding (earlier Outlook

versions)
Mail stays in the Outbox
Mail stays in queues
13
Common causes
Exchange server performance problems
Disk, CPU, or memory bottleneck
Access control list (ACL) conversion problem:
Event ID 9551
Event ID 9552
Missing msExchMasterAccountSID: Event ID
9548
Associated external account right missing on
disabled account
Knowledge Base (KB) article: “How to troubleshoot
public folder performance issues that are related to
ACL conversions in Exchange 2000 and in
Exchange 2003”
http://support.microsoft.com/default.aspx?kbid=328880 14
Common causes (2)

(continued)
Incorrect SiteFolderServer attribute value
Virtual memory fragmentation: Event 9582
KB article: “How to troubleshoot virtual memory
fragmentation in Exchange Server 2003 and
Exchange 2000 Server”
http://support.microsoft.com/default.aspx?kbid=325044
Large number of items in mailboxes
Incorrect antivirus configuration
15
Common causes (3)
(continued)
Faulty hardware
Running online maintenance and backup
during peak hours
Too many cached restrictions, back links, and
searches being put on an individual folder
KB article: “Accessing information store folders may
become slow in Exchange”
KB article: “Clients build new restricted views when
you change item-level permissions”
16
Common causes (4)
(continued)
Third-party server software and COM- add ins
Antivirus, handheld devices, unified messaging,
replication software
Client or add-in problems
Enabling Instant Messaging
Planner options
Incorrect Calendar settings
Incorrect RPC Binding order
Outlook add-ins
17
Common causes (5)
Network issues
Incorrect placement of routers or firewalls
between Exchange and global catalog
Incorrectly configured or faulty hardware
Issues related to Active Directory
performance and other issues
Misconfigured DSAccess
Misconfigured global catalogs
Domain controller (DC)/global catalog
replication
18
Where to start?
Ask questions about the problem
Levels affected
Organization, site, server, local clients, remote clients?
Frequency
How frequent: monthly, weekly, multiple times per
week, daily, multiple times per day?
Is it more frequent in the morning?
What is the client doing when this problem occurs?
Is it always the same users?
19
Where to start? (2)
Ask questions about the problem
Tip: In Outlook 2002, if the server name

shown in the Requesting data dialog box
is in the fully qualified domain name
(FQDN) format, Outlook is waiting for a
response from the directory service. If
you see a short server name, Outlook is
waiting for a response from either the
mailbox server or the public folder
server.
20
Common tools
Collecting data
Microsoft Exchange Server Best Practices

Analyzer Tool (ExBPA)
Microsoft Product Support Reporting Tools
(MPSReports)
Network Monitor (NetMon)
Performance Monitor Wizard (PerfWiz)
Microsoft Exchange User Monitor Tool
(Exmon)
21
Common tools (2)
Exchange Server Best Practices Analyzer Tool
The Exchange Server Best Practices Analyzer
programmatically collects settings and values
from data repositories such as Active
Directory, the registry, the metabase, and
Performance Monitor.
After the data is collected, a set of
comprehensive ‘best practice’ rules are
applied to the topology.
Administrators who run this tool receive a
detailed report of the recommended changes
that can be made to the environment to
achieve better performance, scalability, and
uptime. 22
Common tools (2)
Exchange Server Best Practices Analyzer Tool
ExBPA Web sites

http://www.microsoft.com/exchange
/downloads/2003/exbpa/default.asp
www.exbpa.com
Exchange ExBPA blog
http://blogs.msdn.com/exchange/archive
/2004/09/21/232516.aspx
23
Common tools (3)
Network Monitor
Collects network packet information

The key is concurrent traces from the server and
a client computer
Keys to a good capture
Increase the log buffer
Make sure that % Buffer Utilized does not reach 100
percent
Make sure to get ANY to ANY traffic, and do not use
filters
Have a list of the IP addresses in the trace
24
Common tools (4)

(MPSReports) gathers critical system and
logging information that is used in
troubleshooting support issues.
MPSReports does not make any registry
changes or modifications to the operating
system.
There are eight specialty versions. Each version
gathers some of the same basic information, but
there are specific reports unique to each support
scenario category. See the Readme.txt files for
more information about each version.
25
Common tools (5)
Microsoft Product Support Reporting Tools, continued
MPSReports for Exchange creates a .cab file

<computername>_MPSReports_<timestamp>.CAB
Stored in %systemroot%\MPSReports\Exchange\cab\
Download MPSReports
http://www.microsoft.com/downloads/details.aspx
?familyid=cebf3c7c-7ca5-408f-88b7-f9c79b7306c0
&displaylang=en
26
Common tools (6)
Performance Monitor Wizard
The Performance Monitor Wizard (PerfWiz) simplifies the

process of gathering performance monitor logs. It
configures the correct counters to collect, sample
intervals, and log file sizes.
PerfWiz can create logs for troubleshooting operating
system or Exchange server performance issues.
With this tool, you can walk through a series of wizard-
based dialog boxes and answer some questions. Then
PerfWiz handles configuring the Perfmon logs so that you
never even have to open Perfmon.
Download PerfWiz
http://www.microsoft.com/downloads/details.aspx?FamilyID
=31fccd98-c3a1-4644-9622-faa046d69214&DisplayLang=en
27
Common tools (7)
Microsoft Exchange User Monitor Tool
The Microsoft Exchange User Monitor (ExMon)

tool lets administrators view and evaluate
individual users' usage and experience with
Exchange Server.
Administrators can gather real-time data that
helps them better understand current client
usage patterns and plan for future use.
Download Exmon
http://download.microsoft.com/download/3/5/b
/35b64e21-44b6-4d30-b4d2-3a41f3635e7e
/Exmon.msi
28
Analyzing data
ExBPA
ExBPA performs a proactive health check on the
Exchange environment to make sure that the
configuration and the deployment are correct.
Some errors will be clear and marked by the
ExBPA rule set. You may have to manually look
for other errors.
By default, ExBPA does not detect topology
configuration errors, but it collects enough
information for you to find these errors.
ExBPA collects detailed information about a
deployment and looks for well-known
misconfiguration issues or other problems. You
can use this information to help diagnose and
correct the problem. 29
Analyzing data (2)
ExBPA
30
MPSReports
What are we looking for?
Event IDs 700 - 704: Online maintenance events
Exchange Server online maintenance should complete
at least one time a week.
KB article: “Understanding performance and scalability
characteristics of Exchange 2000 MDB online
maintenance”
Event ID 1221: Online defragmentation event
If white space is excessive, run offline defragmentation
using eseutil /d to avoid backing up unnecessary
white space and save valuable time.
KB article: “Cannot determine free space available in
Exchange database”
31
MPSReports (2)
Event IDs 118 and 474: ESE/ESE98 -1018 errors

Filter for these events, and then search the description
for “-1018”. If backups are completing successfully,
there should be no -1018 errors.
KB article: “Understanding and analyzing -1018,
-1019, and -1022 Exchange database errors”
http://support.microsoft.com/default.aspx?kbid
=314917
KB article: “Event ID 474 error indicates a hardware
failure”
http://support.microsoft.com/default.aspx?kbid
=327334
32
MPSReports (3)
Any evidence of known issues?
Events 9 and 11: Disk subsystem errors
These events indicate hardware problems with
either the controller or, more likely, a device
that is attached to the controller in question.
KB article: “How to troubleshoot Event 9 and
Event 11 error messages”
http://support.microsoft.com/default.aspx?
kbid=154690
If you are troubleshooting a DC replication
problem that will eventually become an
Exchange problem, use the MPSReports
version created for domain controllers. 33
MPSReports (4)
Evidence of known issues, continued
Notable DSAccess events

2060 (DSAccess) – No global catalog server was
found on the local site.
2102 (DSAccess) – DSAccess was not able to find
any domain controllers (DCs) suitable for Lightweight
Directory Access Protocol (LDAP) queries.
2103 (DSAccess) – DSAccess was not able to find
any global catalogs suitable for LDAP queries.
2064 (DSAccess) – The DSAccess process was not
able to contact any DC in the same domain as the
Exchange 200x server and was not able to contact
any global catalog in the same forest. This indicates
an unstable network.
34
MPSReports (5)
Evidence of known issues, continued
More notable DSAccess events

9057 (MSExchangeSA) – The Name Service
Provider Interface (NSPI) proxy cannot contact
any global catalogs that support the NSPI
service. New clients will be refused until a
global catalog is available.
9176 (MSExchangeSA) – The NSPI proxy
can contact a global catalog, but the global
catalog does not support the NSPI service.
After a DC is changed to a global catalog, the
global catalog must be rebooted to support
MAPI clients.
35
MPSReports (6)
System Information
Verify what drivers are on the computer by viewing System
Summary, Components, Network, Adapter.
If the drivers are not up to date, update to the latest drivers by
visiting the manufacturer’s Web site and searching for the specific
device.
Loaded Modules
Verify what modules are loaded by viewing System Summary,
Software Environment, Loaded Modules.
You can quickly confirm all versions of antivirus and antispam
software, backup software, third-party security packages, and
more.
Netdiag
DCDiag
ExchInfo, Boot.ini, Cluster.log
NLtest
36
Performance Monitor
There are four primary performance bottlenecks

that Exchange may experience.
Disk I/O
Processor
Memory
Network
37
What to monitor
Counter Expected value
LogicalDisk*\Average Disk
<40 ms (Database drives)
sec/Read
LogicalDisk\Average Disk
<40 ms (Database drives)
sec/Write
MSExchangeIS\RPC Requests <30
MSExchangeIS\RPC Averaged
<50 ms
Latency
• You can also use PhysicalDisk. Just make sure that you
know what disks are used by databases.
• Troubleshooting Microsoft Exchange Server performance
http://www.microsoft.com/technet/prodtechnol/exchange/g
uides/TrblshtE2k3Perf/4b012bda-8711-4617-9239-f3527d
e884a3.mspx
38
Performance – Disk I/O
Database drives, EDB and STM – random
I/O
Access to both files is random, so keeping
them on separate volumes does not provide
significant performance gains.
We recommend RAID-0+1 or RAID-10 for
high reliability with best read/write
performance.
Public folder data can also be stored on
RAID-5, because this data is typically written
one time and read many times.
39
Performance – Disk I/O (2)
Transaction logs – sequential I/O
Writes to transaction logs are always
sequential.
Reliability is very important. If an outage
occurs, only the transaction logs make
recovery of databases possible.
We recommend RAID-0+1 or RAID 10.
RAID-1 is also acceptable.
Every set of transaction logs should be on a
separate drive. This provides a significant
improvement in performance and fault
tolerance.
40
SMTP queue – sequential I/O
The Simple Mail Transfer Protocol (SMTP)
queue stores messages until they are written
to the database or delivered to another
server.
Duration of storage is very short, so
performance takes precedence over capacity
and reliability.
We recommend RAID-1. RAID-0 gives better
performance, but a trade-off is potential mail
loss.
KB article: “How to change the Exchange
2000 SMTP Mailroot directory location”
30
41
Temp disk and page file disk
We recommend that you use RAID-1 for the
system partition. This includes Exchange
binaries.
The page file can be above RAID-1 or even a
single volume. On a high-use server, we
recommend that you put the page file on a
separate volume.
Always format the file system as NTFS and
select the default block size of 4 kilobytes
(KB).
It is a good idea to change the global
environment variable TEMP to a separate
drive. This TEMP drive is used for all RTF-
HTML conversion. 42
The good…
Smaller spikes can be okay. Look at the trend.
43
…and the bad
This is a simple example; the principle is the same.

Database drives are the number one suspect.
44
Improving disk performance
Enable caching on the array controller
Increase log buffers
Increase the database cache
Align disk partitions with storage track boundaries
Enforce message size limits
Enforce mailbox size limits
45
Performance – Processor
If the Processor(_Total)\% Processor Time

counter is approaching 100 percent, this
indicates a CPU bottleneck.
If the Processor(_Total)\% Processor Time
counter is high, check to see if the
MSExchangeIS\RPC Requests counter is
increasing. If the MSExchangeIS\RPC
Requests counter reaches the maximum of
100, client time-outs occur. The Exchange
store can handle only 100 RPC requests at the
same time.
46
Performance – Processor (2)
What are we looking for, continued
If you have determined that the problem is with

the CPU, you should determine what is using
the CPU resources.
Process(STORE.EXE)\% Processor Time
Process(inetinfo)\% Processor Time
Process(emsmta)\% Processor Time
Process(system)\% Processor Time
47
Performance – Processor (3)
What are we looking for, continued
Processor utilization should never be greater than 80

percent for extended periods of time.
System \Processor Queue Length should be less than 2.
48
Performance – Memory
Store.exe will use as much available

memory as it deems is needed and then
release memory when it is required by
another process.
http://blogs.technet.com/exchange/archive/2004/08/02
/206012.aspx
Physical memory and virtual memory are
different.
Virtual memory fragmentation is not a
memory leak!
49
Performance – Memory (2)
Make sure that you have the /3GB switch in Boot.ini file
(collect with MPS Reports) for Exchange running on
Microsoft Windows® 2000 Server Advanced Edition.
Use the /3gb /userva3030 switch for Exchange running
on any version of Microsoft Windows Server™ 2003.
Memory \Available mbytes counter
Potential problem if available memory is less than 50 megabytes
(MB)
Memory\pages/sec counter
Indicates the rate at which pages are read from or written to disk
to resolve hard page faults
Should be lower than 1,000 at all times
50
Performance – Network
MSExchangeIS\RPC Requests counter
Shows how many MAPI RPC requests are being
serviced by the Exchange store
The Exchange store can service only 100 requests at
a time
MSExchangeIS\RPC Operations/sec counter
Shows the rate at which the Exchange store is
servicing user requests
The key to using these two counters is relatively
simple
If RPC requests are low and RPC operations/sec
(outstanding requests) is zero, the performance
problem is occurring before Exchange processing
occurs 51
Performance – Network (2)
Example of an Exchange problem
52
Example of an Exchange problem
53
Network Interface\Bytes Total/sec

Indicates the rate at which the network adapter is processing
data bytes
Includes all application and file data and protocol information
such as packet headers.
Should be less than 6 - 7 megabytes-per-second (Mbps) for a
100-Mbps network adapter
Network Interface\Packets Outbound Errors
Indicates the number of outgoing packets that could not be
transmitted because of errors
Should be zero (0) at all times
Network Interface\Output Queue Length
Should remain lower than 1 or 2
54
Rule out Active Directory-bound
problems
Exchange depends on Active Directory.
Look for bottlenecks on your Active Directory servers:
CPU bottlenecks, disk bottlenecks, insufficient memory.
Most techniques to identify problems with Exchange are
equally useful to identify problems with Windows Active
Directory.
Blog entry: “Overloaded GC, perhaps?”
http://blogs.msdn.com/mfp2/archive/2004/03/19/92717.aspx
Blog entry: “Outlook’s Requesting Data pop-up box”
http://blogs.msdn.com/mfp2/archive/2004/03/08/86204.aspx
Segment inter-server and global catalog server traffic
Check for OAB downloads
Use Exmon to identify top bandwidth users

55
Improving Active Directory
performance
Offload distribution lists and expand query-based distribution lists to

dedicated global catalog servers and Exchange servers
Limit distribution list size and use nested distribution lists
Verify good network connectivity
Check request latencies
56
Best practices
Outlook and Exchange
How to troubleshoot the RPC Cancel Request
dialog box in Outlook 2003 or in Outlook 2002
Description of the support process for issues that
are related to the "Outlook is retrieving data"
message in Outlook 2002 and Outlook 2003
You receive an "Outlook is retrieving data"
message repeatedly in Outlook 2003
How to troubleshoot public folder performance
issues that are related to ACL conversions in
Exchange 2000 and in Exchange 2003
http://support.microsoft.com/default.aspx?kbid=328880 57
Best practices (2)
OL2002: You receive the "Cancelable RPC"
dialog box when you use the meeting planner
options
How to troubleshoot missing and duplicate
appointments in Outlook
How to change the Cancel Request dialog box
behavior in Outlook 2002
How to change the behavior of the Cancel
Request dialog box in Outlook 2003
58
Best Practices (3)
Outlook meeting requests: Essential do’s
and don’ts
http://office.microsoft.com/en-us/assistance
/HA011276781033.aspx
http://office.microsoft.com/training/training.aspx
?AssetID=RC011659751033
Evaluating with regards to hardware
http://www.microsoft.com/technet/prodtechnol/exchange
/EXBPA/a59de247-8f05-419d-a4ea-fc4973788863.mspx
Recommended mailbox size limits and disk
sizing
http://blogs.technet.com/exchange/search.aspx?q=
%22nicole+allen%22&p=1
59
Best practices (3)
Requesting data from the Exchange server...' or 'Outlook

is trying to retrieve data…' client messages
http://blogs.technet.com/exchange/archive/2005/05/25/405353.aspx
Large messages affect server performance
http://blogs.msdn.com/jeremyk/archive/2004/02/29/81773.aspx
Antivirus articles
Exchange Server 2003 Technical Documentation Library
http://www.microsoft.com/technet/prodtechnol/exchange/2003/library
/default.mspx
60
Questions
61
Thank you for joining us for today’s event.
For information about all upcoming Support WebCasts, and access

to the archived content (streaming media files, PowerPoint® slides,
and transcripts), visit the Support WebCast site at
http://support.microsoft.com/WebCasts/
We sincerely appreciate your feedback. Please submit any comments

or suggestions about the Support WebCasts on the “Contact Us”
page of the Support Web site at
http://support.microsoft.com/servicedesks/webcasts/feedback.asp.
© 2005 Microsoft Corporation. All rights reserved.

This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

How To Analyze and Troubleshoot The Cancelable RPC Dialog Box

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

How To Analyze and Troubleshoot The Cancelable RPC Dialog Box

Uploaded by

Copyright:

Available Formats

How to analyze and

troubleshoot the Cancelable

Nagesh Mahadev and Hamza Hassen

Microsoft Outlook® 2000 SR-1 and earlier versions

All requests to the

Stores the name of

Logon: contacts GC1

Client stops responding (earlier Outlook

Exchange server performance problems

Tip: In Outlook 2002, if the server name

Microsoft Exchange Server Best Practices

ExBPA Web sites

Collects network packet information

Microsoft Product Support Reporting Tools

MPSReports for Exchange creates a .cab file

The Performance Monitor Wizard (PerfWiz) simplifies the

The Microsoft Exchange User Monitor (ExMon)

Event IDs 118 and 474: ESE/ESE98 -1018 errors

Notable DSAccess events

More notable DSAccess events

There are four primary performance bottlenecks

Smaller spikes can be okay. Look at the trend.

This is a simple example; the principle is the same.

Enable caching on the array controller

Increase log buffers

Increase the database cache

Align disk partitions with storage track boundaries

Enforce message size limits

Enforce mailbox size limits

If the Processor(_Total)\% Processor Time

If you have determined that the problem is with

Processor utilization should never be greater than 80

Store.exe will use as much available

Network Interface\Bytes Total/sec

Segment inter-server and global catalog server traffic

Check for OAB downloads

Use Exmon to identify top bandwidth users

Offload distribution lists and expand query-based distribution lists to

Limit distribution list size and use nested distribution lists

Verify good network connectivity

Check request latencies

Requesting data from the Exchange server...' or 'Outlook

For information about all upcoming Support WebCasts, and access

We sincerely appreciate your feedback. Please submit any comments

© 2005 Microsoft Corporation. All rights reserved.

You might also like