Troubleshooting Virtual

Memory Fragmentation on
Exchange 2000 Servers

Nino Bilic
Technical Lead
Enterprise Messaging Support
Microsoft Corporation
Agenda
 What happens and why?
 Why is this a problem and when?
 Sample events
 What can contribute to the problem?
 Incorrect use of the /3GB switch
 Virus scanning
 Resolving the problem
 If all else fails…data collection

2
What Happens and Why?
 As Exchange database engine works, it uses
virtual memory dynamically.
 Other processes running on the server use
virtual memory too.
 Because of the swapping of memory
involved, the virtual memory gets
fragmented.
 When we reach a certain level of
fragmentation, we start logging events.
 Logging is functionality added in SP1.

3
What Happens and Why? (2)

 It is important to understand that we are not

running out of memory on the server.
 Virtual memory is being fragmented.
 Exchange 2000 will use virtual memory no
matter how much physical memory (RAM) the
server has.

4
Why Is This a Problem and When?

 The Exchange database engine needs 16 MB

of contiguous virtual memory space to fail
over in a cluster environment.
 If we do not have contiguous 16 MB available,
the failover process will fail.
 Generally, this is a problem on clustered
servers only.
 Events might be seen on non-clustered
servers too.

5
Sample Events
 Warning events (logged when less than 32 MB of
contiguous virtual memory space is available):

Event Type: Warning

Event Source: MSExchangeIS
Event Category: Performance
Event ID: 9582
Date: 1/1/2002
Time: 12:00:00 A.M.
User: N/A
Computer: SERVERNAME
Description: The virtual memory necessary to run your
Exchange server is fragmented in such a way that
performance may be affected. It is highly
recommended that you restart all Exchange services
to correct this issue. 6
Sample Events (2)
 Error events (logged when less than 16 MB of
contiguous virtual memory space is available):

Event Type: Error

Event Source: MSExchangeIS
Event Category: Performance
Event ID: 9582
Date: 1/1/2002
Time: 12:00:00 A.M.
User: N/A
Computer: SERVERNAME
Description: The virtual memory necessary to run your
Exchange server is fragmented in such a way that
normal operation may begin to fail. It is highly
recommended that you restart all Exchange services
to correct this issue. 7
Sample Events (3)
 If a 9582 warning is being logged, the failover
on a cluster server will still work.
 If a 9582 error is being logged, the failover on
a cluster server will not work, and as a result
the Exchange virtual server might go down
(service interrupted).
 In most cases, 9582 events will be
transparent to Exchange clients (unless
server is running out of memory, too–slow
response and memory errors).
8
What Can Contribute to the
Problem?
Incorrect use of the /3GB switch

 Windows 2000 can address 4 GB of virtual

address space.
 By default:
 User-mode applications get 2 GB to work in
 Kernel mode gets 2 GB to work in

9
What Can Contribute to the
Problem? (2)
Incorrect use of the /3GB switch (continued)
 When the /3GB switch is used in Boot.ini:
On Windows 2000 Server:
 User mode 2 GB + Kernel mode 1 GB
On Windows 2000 Advanced Server and
Windows 2000 Datacenter Server:
 User mode 3 GB + Kernel mode 1 GB

 Using the /3GB switch on Windows 2000

Server takes 1 GB away from system without
giving it to user-mode applications!
10
What Can Contribute to the
Problem? (3)
Store virus scanning

 Depending on the load, additional store tasks

like virus scanning can fragment the memory
enough to start generating 9582 events.
 This can contribute only if the antivirus
software runs on the Exchange 2000 server.

11
Resolving the Problem

 Absolutely the first thing to do is apply

Exchange 2000 SP3.
 SP3 improvements:
 Top-down memory allocation (Q324118)
 Certain store memory values have been tweaked
 Several other store fixes rolled up (Q315771 for
example)
 In most cases, application of Exchange 2000
SP3 will stop virtual memory fragmentation
problems.
12
Resolving the Problem (2)

 Use the latest available Scsiport.sys driver.

 Windows 2000 SP3 or Q311551 driver hotfix
(post-SP2 for Windows 2000).
 Resolves some known memory
fragmentation problems.

13
Resolving the Problem (3)
 Edit the following registry key (only if the
server has more than 1 GB of RAM!):
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
\Session Manager\HeapDeCommitFreeBlockThreshold=262144

(Note that the registry key should be entered as one line; it is

wrapped here for readability.)

 The value 262144 is a decimal value (not hex).

 See Q315407 for more information.

14
Resolving the Problem (4)
 Decrease the store database maximum cache
size.
 msExchESEParamCacheSizeMax – set it to
the value of “196608” (768 MB).

 See Q266768 for more info about this value –

decreasing this might help if you’re running
into virtual memory problems.

15
If All Else Fails…Data Collection
 If 9582 events are still a problem, the best is
to collect additional data and contact Product
Support Services.
 Follow Q317411 to gather the following:
 System Monitor logs (also known as Performance
Monitor logs)
 Event logs
 System information file (WINMSD)
 Virtual address dump file
 Possibly userdump of the store
16
Additional Resources
 Q324118 “XADM: The Extensible Storage
Engine Database Engine Contributes to Virtual
Memory Fragmentation”
 Q315407 “XADM: The
‘HeapDecommitFreeBlockThreshold’ Registry
Key”
 Q266768 “XSTR: How to Modify the Store
Database Maximum Cache Size”
 Q317411 “XADM: How to Gather Data to
Troubleshoot Exchange Virtual Memory
Issues”
 Q266096 “XGEN: Exchange 2000 Requires
/3GB Switch with More Than 1 Gigabyte of
Physical RAM”
17
Thank you for joining today’s Microsoft Support
WebCast.

For information about all upcoming Support WebCasts,

and access to the archived content (streaming media
files, PowerPoint® slides, and transcripts), visit:
http://support.microsoft.com/webcasts/

Your feedback is sincerely appreciated. Please send any

comments or suggestions about the Support
WebCasts to supweb@microsoft.com.