Designing an Exchange 2000/2003

Routing Group Connector Topology

By: Craig Borysowich

Chief Technology Architect
Imagination Edge Inc.
www.imedge.net
Version 3.7

Pg. 1
 Designing a Routing Group Topology

BACKGROUND
Large Exchange 5.5 environments use X.400 connectors to route both mail and replication traffic. Most
implementations have selected a Hub and Spoke design for their X.400 topology. Most servers have an
X.400 connector directly to the hub, but some sites will go through a Bridgehead X.400 connector to the
Exchange routing hub.

OBJECTIVE
As migrations into Exchange 2000/2003 environments increase, we need to design an efficient way to route
mail messages while managing overall governance. The SMTP mail routing is required for the following:

1. Within the Sites (Intra-Site),

2. Inter-Site
3. Internet Mail

Microsoft Exchange 2000 enables great performance improvements with its message routing when
compared to Exchange 5.5. One of the improvements is the ability to split Exchange 5.5 sites into Routing
Groups and Administrative Groups. Routing Groups are based upon network topology and are designed to
enhance the efficiency of message delivery. Exchange 2000 also utilizes SMTP to deliver messages as its
preferred protocol instead of X.400.

It is anticipated that each Exchange site or group of sites will develop a mail routing topology and strategy
in concert with the rest of the Exchange sites in an organization as an integral part of the overall mail
routing design. In cases where an organization has distributed management and administration of their
Exchange Infrastructure, a team should be established to agree to a set of standards and a common design
approach. This team should include members that are familiar with the current LAN/WAN topology,
current Active Directory implementation and Design, and current Exchange 5.5 background for the
organization.

Page. 1
 Designing a Routing Group Topology

Exchange 2000/2003 routing basics

Terminology

Routing Topology
The routing topology denotes the overall interconnections that multiple sites in an Exchange organization
route mail between themselves and with the Internet and other foreign mail systems.

Routing Group (RG)

A Routing Group is the management unit in the Routing Topology focused on systems that are: always
connected; connected via reliable/always available network links; configured for point-to-point connectivity

Administrative Group (AG)

An Administrative Group is an object management unit where the objects that can be managed include:
servers, routing groups, folders, etc. The Administrative Groups can be delegated to manage any grouping
of objects.

So, Administrative Groups are created for managing objects. The Routing Groups are created for managing
the Routing Toplogy. Routing Groups must be created within an Administrative Group – they are
subordinate to an Administrative Group. A single Administrative Group could contain all of the Routing
Groups with other Administrative Groups managing other server and service components. Servers can be
Drag & dropped between Routing Groups.

Routing Group Master (RGM)

Each Routing Group contains a Routing Group Master, which is the first server added to a Routing Group.
The Routing Group Master Role can be changed by an administrator. The Master co-ordinates changes to
the Link State information within the Routing Group.

Link State Information (LST)

Information about the state of messaging routes (links) in an Exchange 2000/2003 messaging system
derived from the link state algorithm to quickly and frequently calculate the state of system links for up-to-
date status about routes. Exchange 2000 servers use link state information to make the best routing choice at
the source rather than sending a message down a path where a link is not working. This eliminates message
bounce and looping.

Bridgehead Server (BH)

The Bridgehead Server is an Exchange server that connects to other Exchange servers using the same
communications protocols so information can be passed from one server to another. In Exchange 2000, a
bridgehead server is a connection point from a routing group to another routing group, remote mail system,
or other external mail system.
Page. 2
 Designing a Routing Group Topology

Routing Group Connector (RGC)

A connector that specifies the connection of a local routing group to a server in a remote routing group. It
also specifies the local bridgehead server, if any, and the connection cost, schedule, and other configuration
properties.

All of the Bridgehead servers in a Routing Group communicate their connectivity information to the
Routing Group Master, but only for changes. The Master then maintains/rebuilds the link state information
and communicates it to all of the servers in the Routing Group. If the Master goes offline, then the servers
continue as normal based on the last report of the Link State information received from the Master. The
normal flow of mail within a Routing Group will also pass Link State Information.

The Connectivity between all servers within a Routing Group is SMTP. For link state updates, Microsoft
uses a proprietary binary data protocol that does not use the SMTP virtual Server. The Microsoft Exchange
Routing Service handles all routing updates within the Routing Group. This service executes within the
INETINFO.EXE and each Bridgehead server binds to the Master on TCP port 691 and vice versa to all
servers in the Routing Group for link state updates. Connectivity between Routing Groups is via a
Bridgehead server.

A Simplified overview of Message routing in Exchange 2000/2003 follows:

Page. 3
Designing a Routing Group Topology

Fig 1: Simplified Message Flow for Exchange 2000/2003

Page. 4
 Designing a Routing Group Topology
The following three routing methodologies should be considered when designing a routing topology:
1. Full Mesh Topology
2. Limited Mesh Topology
3. Modified Hub & Spoke Topology.
TOPOLOGY Pros
Full Mesh 1. While this topology offers greater
efficiency of mail routing, this topology
is highly dependant on the underlying
network topology - If underlying
Network topology is Hub-and-spoke
then mesh does not add as much
efficiency as it could. It does allow for
redundant paths and will bypass the
overhead required of the HUB to route
the messages.).
2. No single point of failure
3. Most redundant topology – mail routes
calculated and delivered based on
OSPF, dynamically modified when
outages in bridgeheads occur thus
taking advantage of performance
characteristics.
Limited Mesh 1. While this topology offers greater
efficiency of mail routing, this
topology is highly dependant on the
underlying network topology - If
underlying Network topology is
Hub-and-spoke then mesh does
not add much efficiency as it could.
It does allow for redundant paths
and will bypass the overhead
required of the HUB to route the
messages).
2. No single point of failure
3. Semi-redundant topology- mail
routed over various paths due to
performance characteristics
Modified Hub- 1. Offers optimum governance in
and- Spoke managing:
a. Virus outbreaks
b. Spam
c. Tracing mail
2. Reduces the number of connectors
from the site bridgehead server to
the hub.
Cons
1. Lacks adequate governance to
manage:
a. virus,
b. spam
c. mail flow trace
d. and other threats to mail
environment.
2. An equal partner in mail flow and
Exchange 2000 from functional
perspective – (Site bridgeheads
could be routing other sites mail if
redundant bridgeheads are not
implemented for each site)
1. Moderate governance to manage:
a. virus,
b. spam
c. mail flow trace
d. and other threats to mail
environment.
2. An equal partner in mail flow and
Exchange 2000 from functional
perspective – (Site bridgeheads
could be routing other sites mail if
redundant bridgeheads are not
implemented for each site)
1. Limited to risk to intra-site and
internet mail flow if both servers
became unavailable. Mail within a
site would not be impacted.
2. Would require a centralized
redundant environment to offer high
service availability.
3. Follows old style 5.5 design models.
Routing improvements in Exchange
2000/2003 should eliminate the
need for this model.
Page. 5
 Designing a Routing Group Topology

ROUTING GROUP TOPOLOGY RECOMMENDATION

Considerations
Whether you are coexisting and migrating from an existing Exchange 5.5 organization or building a new
organization, you should take the following design considerations into account when selecting a topology:

• In native mode, Exchange 2000 can be extremely dynamic because you can change the memberships
of the routing groups and administrative groups. This is useful when administration models or network
topologies change.
• Exchange 2000 provides load balancing in the form of a round-robin DNS between servers, both
sources and targets. A round-robin DNS is a mechanism that directs incoming requests to servers on a
rotating basis. This is done by looping through a list of IP addresses belonging to the servers in the
configuration. When an e-mail client attempts to access a mailbox on an Exchange server, the client is
given the first IP address on the list. The second client request is given the second IP address in the
list, and so on. If there are four servers on the round-robin list, all four IP addresses are used before the
first IP address is used again, and the loop starts over. In addition, Exchange 2000 offers
improvements over the Exchange 5.5 Site Connector if one of the source bridgehead servers is
down—Exchange connectors automatically try not to use that server until it comes back up. If there
are multiple connectors with the same cost, each server picks a random connector and uses it for a
period of time. Over multiple servers, this functionality simulates round-robin behaviour.
• When a client must use an alternate server to access public folder content, Exchange 2000 uses routing
groups to calculate the closest available server, which it determines by using a cost property set on the
Routing Group connector. The cost for each Routing Group connector is stored in a single cost
database that is shared with e-mail routing calculations. Redundant cost tables maintained in previous
versions of Exchange are eliminated. In Exchange 5.5, site affinities were not transitive. For example,
if you establish an affinity between site 1 and site 2, and between site 2 and site 3, you do not
automatically get affinity between site 1 and site 3. In Exchange 2000, affinities are transitive. If all
routing groups in an organization are connected to allow e-mail to flow, all servers receive public
folder referrals. If a server does not contain replicas of public folders, you can mark specific Routing
Group connectors to deny public folder referrals; the client contacts the next server in the referral list.
• Because SMTP is used in a routing group and SMTP is more resilient over slow links, routing groups
can span slow networks more effectively than RPC, which is the native transfer mechanism for
Exchange 5.5. Thus, your organization may be able to deploy fewer routing groups by using Exchange
2000 than by using sites with Exchange 5.5.
• SMTP mail is not compressed; though you reduce network bandwidth requirements with SMTP, the
increased load on the Exchange server CPU would reduce server performance.
• Unlike Exchange 5.5, where intra-site and inter-site RPC communications are encrypted, SMTP
communications in Exchange 2000 are not encrypted. However, each server uses SMTP
authentication with Kerberos. There are two options for encryption, although neither is done by
default: Internet Protocol security (IPSec), which is built into Microsoft Windows 2000, and Transport
Layer Security (TLS) encryption, which is built into the SMTP service.
• Upgrading an Exchange 5.5 hub, in a hub-and-spoke design, can significantly increase hub
performance and stability because performance is enhanced by Exchange 2000 and the use of SMTP.
• Link state routing is most effective in an organization with multiple routing groups using multiple
paths. In a traditional hub-and-spoke design, you do not see as much improvement as you do in a mesh
network or a network with multiple connections.
Page. 6
 Designing a Routing Group Topology
• Adding routing groups increases the size of the link state table and increases potential link state status
messages. This can increase the amount of link state message traffic.

Exchange no longer includes a Dynamic RAS connector. Instead, Exchange 2000/2003 uses the on-demand
capabilities in Windows 2000/2003, specifically, the combination of the Routing Group connector and the
Routing and Remote Access demand-dial interface.

Exchange 2003 Specifics

Microsoft highlights that Exchange 2003 further improves upon the link state routing features of
Exchange 2000 by reducing the amount of link state traffic in two ways.
• Performance is improved over what are commonly known as oscillating links, or links that are intermittently available and
unavailable. Exchange 2003 reduces link state traffic by attempting to determine if the connector is oscillating. If there are
multiple conflicting state changes in the link state queue for a connector in a given interval, the connector is considered an
oscillating link and its link state remains up (in service or available for that period). Leaving an oscillating connector up
rather than continually changing the link state reduces the amount of link state traffic that is replicated between servers.
• Performance is improved in the case of a site to which there is only one route. In this case, Exchange 2003 reduces link
state traffic by determining that no alternate path exists and suppressing the link state information. If no alternate path
exists for a link, the link state is always marked as up (in service). Exchange lets mail queue for delivery and sends it when
the route becomes available.
Both of these changes enhance performance because they reduce the propagation of link state information.
Other than the above, there is not much in the Exchange 2003 product that would really impact your routing
topology decision.

Routing Group Topology Preference

In evaluating the various topologies, an organization should prioritize moving to the full-mesh topology.
Scaling back to the limited mesh topology if necessary, and then falling back to a 5.5 style hub/spoke
topology only in the most dire of circumstances. Any perceived benefits in the hub/spoke topology to aid in
combating virus or spam outbreaks are essentially mythical. In a properly designed and managed Exchange
2000/2003 environment, you should be able to halt routing in the mesh faster and more easily than in a
hub/spoke implementation.

Final Notes on Routing Group Connection Options

Assuming that the Full Mesh Routing Group Topology that is recommended above is accepted, we must
now look at how the Routing Group(RG) Connectors are going to be configured.

In current X.400 connection topologies, sites have either a single or a redundant connection with the Hub, if
this connection goes down, outbound messages from the site are queued locally until the connection is back
up. As well inbound messages to the site are queued at the hub until the connection again becomes
available. This configuration has a single point of failure to each site as well as a single point of failure at
the hub. Full Mesh topology removes the need for running a hub as a single point of failure and allows for
mail to find an operation route through a myriad of avenues. Remember that e-mail is a communications
tool and that all parties within all Exchange sites should want mail to get to its destination. Even if that
means that a few microseconds of CPU time has to be spent forwarding mail between two sites that have
lost a direct connection.

Page. 7
Designing a Routing Group Topology

Fig 2: Detailed Exchange Message Flow Diagram

Page. 8
 Designing a Routing Group Topology

Planning Routing Groups

A routing group must consist of servers in the same forest. In fact your entire Org must be contained in the
same forest. Major mail routing issues will occur if you try to split an Org between multiple forests.

A permanent connection must exist between all servers participating in the same Routing Group. Servers
that are on intermittent or on-demand connections must be isolated in their own Routing Group. All servers
within the routing group must be able to contact the Routing Master.

When you use multiple Routing Groups, you need to have connectors (i.e. bridgeheads) between two
servers in those routing groups. All mail traffic being routed will go in and out through the bridgehead.
You can have multiple servers in a Routing Group acting as bridgeheads to other Routing Groups.
Exchange 2000/2003 will load balance the connections after a period of time and if one connection goes
down, Exchange will re-select an operational route.

In most cases the Routing Groups should be connected and configured as a Routing Group Connector. In
cases where you are routing mail to through the Internet, then you connectors should be configured as
SMTP. Only when routing to a legacy X.400 Administrative Management Domain (ADMD) or meta-
directory should you use the X.400 Routing Group Connector configuration.

Configuring the Routing Group Connector in RGC mode is protocol independent with SMTP native mode
and RPC available for 5.5 co-existence. The connector is Directory aware for the Remote Bridgehead
server.

Securing Mail Routing

Security for routing mail can be a big issue for many organizations. Legacy X.400 connections were
obfuscated, but not encrypted. SMTP connections are not encrypted, but also obfuscated using Transport
Neutral Encapsulated Format.

There are two ways to secure mail routing: SMTP encryption/authentication; or using IPSec. It is highly
recommended to look at using IPSec for transport security between Exchange Servers. Not only with this
secure and encrypt your mail being routed between sites, but it will secure all IP-based communications
between your Exchange servers.

IPSec can play one of three Roles on a server:

-Basic IP port filtering - like a firewall on the server

-Authenticating Hosts before and during communications
-Encrypting data exchanged between servers

IPSec can also perform all three of these functions on a server at the same time.

Page. 9