Technote: AIX EtherChannel Load Balancing Options http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

AIX EtherChannel Load Balancing Options

Document ID:
Document Author: Cindy K Young TD101260
Additional Author(s):
Jorge R Nogueras
Doc. Organization:
Advanced Technical Support Document Revised: 10/28/2003
Product(s) covered: AIX; Ethernet; pSeries; RS/6000; TCP/IP

Abstract: This article is an overall view of the user-configurable AIX V5 EtherChannel load
balancing behavior options, including the new Hash Mode parameter.

AIX EtherChannel Load Balancing Options

This article is an overall view of the user-configurable AIX EtherChannel load balancing behavior
options, including the new Hash Mode parameter.

Introduction. A new SMIT menu option called "Hash Mode" appears when AIX users install software APARs
available in November, 2003. The menu gives standard mode EtherChannel users the option of distributing
outgoing traffic based upon TCP or UDP port values. Software support is provided in APAR IY48331 for AIX
V 5.1 and APAR IY48170 for AIX V 5.2

Load balancing behavior. Two SMIT-selectable load balancing methods for outgoing traffic are offered --
round robin, which spreads the outgoing traffic evenly across all the adapters in the EtherChannel; and
standard, which selects the adapter by algorithm. The Hash Mode parameter, shown in the following SMIT
configuration menu, determines the numerical value used in the algorithm.

Change / Show Characteristics of an EtherChannel / Link Aggregation

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

EtherChannel / Link Aggregation ent6

EtherChannel / Link Aggregation Adapters ent1,ent2,ent4,ent5
Backup Adapter NONE
Delete Backup Adapter
Add Main Adapter
Delete Main Adapter
Add Backup Adapter
Enable Alternate Address no
Alternate Address [0x000000000000]
Enable Gigabit Ethernet Jumbo Frames no
Mode standard
Hash Mode default
Internet Address to Ping []
Number of Retries [3]
Retry Timeout (sec) [1]
Apply change to DATABASE only no

The following chart summarizes the valid load balancing option combinations offered in the SMIT screen.

1 of 5 12/10/04 10:27 AM
Technote: AIX EtherChannel Load Balancing Options http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

HASH OUTGOING TRAFFIC DISTRIBUTION

MODE
MODE (across adapter ports within the EtherChannel)
The traditional AIX behavior. The adapter selection algorithm uses the last byte
standard of the destination IP address (for TCP/IP traffic) or MAC address (for ARP and
default
or 8023ad other non-IP traffic). This mode is typically the best initial choice for a server
with a large number of clients.
The outgoing adapter path is selected via algorithm using the combined source
and destination TCP or UDP port values. Average the TCP/IP address suffix
standard
values in the "Local" and "Foreign" columns shown by netstat -an command.
or src_dst_port
Since each connection has a unique TCP or UDP port, the three port-based
8023ad
hash modes provide additional adapter distribution flexibility when there are
several, separate TCP or UDP connections between an IP address pair.
The adapter selection algorithm uses the source TCP or UDP port value. In
standard
src_port netstat -an command output, the port is the TCP/IP address suffix value in the
or 8023ad
"Local" column.

standard The outgoing adapter path is selected via algorithm using the destination
or 8023ad dst_port system port value. In netstat -an command output, the TCP/IP address suffix
in the "Foreign" column is the TCP or UDP destination port value.
Outgoing traffic is spread evenly across all the adapter ports in the
round
default EtherChannel. This mode is the typical choice for two hosts connected
robin
back-to-back (i.e. without an intervening switch).

Round Robin. All outgoing traffic is spread evenly across all of the adapters in the EtherChannel. It provides
the highest bandwidth optimization for the AIX server system. While round robin distribution is the ideal way
to utilize all the links equally, AIX administrators should consider that it also introduces the potential for
out-of-order packets at the receiving system.

Standard algorithm. The standard algorithm is used for both standard and IEEE 802.3ad-style link
aggregations. AIX divides the last byte of the "numerical value" by the number of adapters in the
EtherChannel and uses the remainder to identify the outgoing link. If the remainder is zero, the first adapter
in the EtherChannel is selected; one means second adapter, and so on. (The adapters are selected in the
order they are listed in the SMIT menu.)

The "Hash Mode" selection determines the numerical value used in the calculation. By default, the last byte
of the destination IP or MAC address is used in the calculation, but the source and/or destination TCP or
UDP port values may also be utilized. These alternatives allow the AIX administrator to fine-tune the
distribution of outgoing traffic across the real adapters in the EtherChannel.

In default standard mode, the adapter selection algorithm is applied to the last byte of the destination IP
address for IP traffic. For ARP and other non-IP traffic, the same formula is applied on the last byte of the
destination MAC address. Unless there is an adapter failure which causes a failover, all traffic between a
host pair in default standard mode goes out over the same adapter. For IP traffic, the port selected could be
changed by selecting a different hash mode value or by implementing a different IP address (perhaps via IP
aliases) at the target system.

In src_dst_port hash mode, the source and destination port values are added, then divided by two. The
resultant whole number (no decimals) is plugged into the standard algorithm. TCP or UDP traffic is sent on
the adapter selected by the standard algorithm and selected hash mode value. Non-TCP or UDP traffic will
fall back to the default algorithm, meaning the last byte of either the destination IP or MAC address.

Incoming traffic distribution. Remember that the AIX adapter selection applies to outgoing packets only.
The packets incoming to the AIX system are governed by the algorithm configured at the switch. The traffic
distribution schemes used by the two components need not match -- the algorithms can be (and often are)
different. For example, the switch might direct packets based on the XOR of source and destination MAC
addresses. In this case, the destination MAC address is the same for packets going to the AIX system, but
the different source address causes the packets to be sent over different adapters.

Understanding packet distribution. Curious about how packets are being distributed over the ports on your
system? Use the entstat or netstat command to view aggregate statistics of all the adapters in the
EtherChannel. For example, netstat -v ent6 reports the key information about EtherChannel device ent6.
Statistics for outgoing packets are a result of AIX's configuration. If round robin is configured, for example,

2 of 5 12/10/04 10:27 AM
Technote: AIX EtherChannel Load Balancing Options http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

statistics will be nearly identical for each underlying adapter. The incoming packet statistics reflect the
network switch configuration. Use the netstat -an command to view the TCP or UDP source and destination
port address values.

Predicting adapter utilization -- Example Environment. The effects of different hash mode values on the
standard adapter selection algorithm are best illustrated in an example. In this fictitious environment, the
EtherChannel (ent6) is configured as IP address 10.9.8.1 (the en6 interface) and is comprised of individual
adapters ent1, ent2, ent4, ent5. To demonstrate the adapter selection, the example system communicates
with six systems with sequential IP addresses ranging from 192.26.91.100 through 192.26.91.105.

The outgoing adapter port is selected via the mathematical remainder of the calculation hash mode
numerical value / the number of real adapters in the EtherChannel. There are four adapters in this
EtherChannel, so the numerical value is always divided by four. The numerical value changes from the last
byte of the destination IP address to the source port, then to the destination port, and then to the average
of the two port values in the four Hash Modes. It is this change in numerical value that causes the variation
in the adapter selection.

The result of the calculation is the mathematical remainder. If the remainder is zero, the first adapter in the
SMIT list is selected. In the example, this is adapter ent1. If the remainder is three, the fourth adapter (ent5
in our sample) is selected, and so forth. The complete matchup of the remainders, the selection order, and
the actual adapters in the example are shown in the following figure.

Adapter Actual
Remainder
in list Adapter
0 1st ent1
1 2nd ent2
2 3rd ent4
3 4th ent5
Figure 1: Adapter selection in the sample environment.

The following partial netstat -an command output shows IP addresses and TCP/UDP ports for nine active
connections the local system might have with the remote systems 192.26.91.100 through 192.26.91.105. In
this case, the "Local Address" column reflects the EtherChannel's 10.0.8.1 IP address and the "Foreign
Address" column reflects the IP address of the remote system. The numerical values appended to these
local and foreign IP addresses are the source and destination ports, respectively, used in three of the four
hash modes.

# netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address
udp4 0 0 10.9.8.1.36156 192.26.91.105.6000
tcp4 0 0 10.9.8.1.23 192.26.91.102.44070
tcp4 0 0 10.9.8.1.23 192.26.91.103.44075
tcp4 0 0 10.9.8.1.23 192.26.91.105.44078
tcp4 0 0 10.9.8.1.23 192.26.91.101.44078
tcp4 0 0 10.9.8.1.23 192.26.91.104.44089
tcp4 0 0 10.9.8.1.23 192.26.91.100.32776
tcp4 0 3 10.9.8.1.23 192.26.91.100.32777
tcp4 0 0 10.9.8.1.23 192.26.91.104.44411

The following chart shows the mathematical result (the remainder) of applying the four hash modes to the
standard algorithm. Check the remainder value in the Figure 1 chart to identify the real outgoing adapter
selected.
Default src_port dst_port src_dst_port
Line
# Address Address Address Address
remainder remainder remainder remainder
value value value value*
1 105 1 36156 0 6000 0 42156 / 2 2
2 102 2 23 3 44070 2 44093 / 2 2

3 of 5 12/10/04 10:27 AM
Technote: AIX EtherChannel Load Balancing Options http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

3 103 3 23 3 44075 3 44098 / 2 1

4 105 1 23 3 44078 2 44101 / 2 2
5 101 1 23 3 44078 2 44101 / 2 2
6 104 0 23 3 44089 1 44112 / 2 0
7 100 0 23 3 32776 0 32799 / 2 3
8 100 0 23 3 32777 1 32800 / 2 0
9 104 0 23 3 44411 3 44434 / 2 1
* here the address values are first averaged (added, then divided by two). The resultant whole number
value (excluding decimals) is divided by the four adapters in the example environment.
Figure 2. Comparison of results: Standard adapter selection algorithm with various hash mode options.
Observations. In the default hash mode, the numerical value used in the calculation is the last byte in the
destination system IP address. Because this number is the same for all connections to the same destination
IP address, all traffic between a host pair in standard mode goes out over the same adapter. Using this hash
mode, traffic will be fairly well spread over the adapters if the random nature of the IP addresses result in an
even selection and if all of the remote systems do generally the same things (meaning, for instance, one IP
address does not have a huge download).

In this example, the traffic is unbalanced when using src_port hash mode. Traffic shown on line 1 is sent out
adapter ent1 and traffic shown on lines 2-9 will be sent over adapter ent5. Adapters ent2 and ent4 are not
used. Why? The source port address value is 23 (the well-known port for telnet) on all of these connections,
hinting at the way the connection was established. This indicates that for the connections on lines 2-9 the
remote system telnet'ed to the local system.

In dst_port hash mode, the outgoing traffic is fairly well balanced across the adapters in the EtherChannel.
If, however, the local system established the connections, you might see the same results skew seen in the
src_port hash.

In src_dst_port hash mode, the averaging of the fields seems to balance the outgoing traffic fairly well. The
skew in values disappears. So, regardless of the direction in which the connection is initiated, the resulting
values distribute across the adapter ports.

Considerations. The single most important consideration when choosing a load balancing method is nature
of the traffic that is being sent across the EtherChannel. Yet, the final choice will probably be a trade-off
that provides the best distribution for a subset of this traffic. Depending upon the environment, one
administrator may choose to accommodate the majority of the traffic while another opts to design for the
most important form of traffic. There is no single choice that will yield a perfect distribution all the time
because the type of traffic being handled by the typical EtherChannels varies. It may, for example, handle
mixtures of streaming TCP traffic one moment, some ARP traffic the next, then a huge file transfer followed
by short-lived UDP exchanges.

Administrators can use these key questions begin the process of selecting the mode of operation and hash
mode best suited for an environment. Look for patterns and predictability.
Network attachment. Is the local system connected to an Ethernet switch? Is it connected directly
(back-to-back) with another host?
Direction of connection establishment. Do remote systems typically connect to the local host? Is the
local system a server for client systems? Does the local host establish connections to other hosts?
IP addresses. Look for IP address patterns. Does the local host have a few, but long-lived TCP
connections with the same remote hosts? Conversely, does it have many, short-lived connections to
a variety of remote hosts?
Traffic type. The main traffic load may be TCP/IP, but ARP and other non-IP traffic is distributed
differently in standard mode. Is there any other significant type of traffic? Is there a key application
that generates a predictable load?
Conclusions. The results of any mode can be skewed if the numeric values are skewed. In fact, the load
balancing options have been added since the initial AIX V4.3.3 EtherChannel implementation for just this
reason. There is no one-size-fits-all solution to unique environments. The trick is to choose a mode that
results in varied numeric values so that the load is distributed across all of the adapters in the
EtherChannel.

Generally speaking, round robin is ideal for back-to-back connections running jumbo frames. In this
environment, there is no intervening switch, so there is no chance that processing at the switch could alter

4 of 5 12/10/04 10:27 AM
Technote: AIX EtherChannel Load Balancing Options http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

the packet delivery time, order, or adapter path. On this direct cable network path, the receiver sees
packets exactly as sent. Jumbo frames (9000 byte MTU) always yield better file transfer performance than
traditional 1500 byte MTUs. In this case, however, they add another benefit. These larger packets take
longer to send so it is less likely that the receiving host would be continuously interrupted with out-of-order
packets.

Round robin mode can be implemented in other environments but at increased risk of out-of-order packets at
the receiving system. This risk is particularly high when there are few, long-lived, streaming TCP
connections. When there are many such connections between a host pair, packets from different
connections could be intermingled, thereby decreasing the chance of packets for the same connection
arriving out-of-order. Check for out-of-order packet statistics in the "tcp" section of the netstat -s command
output. A steadily-increasing value indicates a potential problem in traffic sent from an EtherChannel.

If out-of-order packets are a problem on a system that must use traditional Ethernet MTUs and must be
switch attached, try the various hash modes offered in standard mode operation. If no particular hash mode
value is obvious, administrators may try each of them, testing and analyzing the results before moving to
the next. Each mode has a particular strength but the "default" and "src_dst_port" modes are the logical
starting points as they are more widely applicable.

The default hash mode works well when the local host establishes connections to many different IP
addresses. It is a good first choice for most busy systems because the random nature of IP address traffic
tends to distribute the traffic in an acceptable manner. If the local host establishes lengthy connections to
few IP addresses, look at one of the TCP or UDP hashing options.

The "src_dst_port" hash mode option is probably the best initial choice because considers both the source
and the destination TCP or UDP port values. In this mode, all of the packets in one TCP or UDP connection
are sent over a single adapter so they are guaranteed to arrive in order, but the traffic is still spread
because connections (even to the same host) may be sent over different adapters. The results of this hash
mode aren't skewed by the connection establishment direction because it uses both the source and
destination TCP or UDP port values.

Use the "src_port" or "dst_port" hash mode options if port values change from one connection to another and
the "src_dst_port" option is not yielding a desirable distribution. For instance, if the local host is an FTP
server, the source TCP or UDP port will be more or less constant across connections. The netstat -s
command would reflect 21, which is the FTP well-known port value. In this case, the "dst_port" hash mode
option would be a better choice. If, on the other hand, the local host primarily establishes connections to
other machines, the source TCP or UDP port values will be random, making the "src_port" hash mode a
better option.

Classification: Category:
Software Planning and Design
Platform(s): RS/6000 or IBM eServer pSeries
O/S: Keywords:
AIX AIX EtherChannel, NIB, network
interface backup, load, balancing,
traffic, SMIT, configuration,
methods, options, hash mode,
standard, algorithm

5 of 5 12/10/04 10:27 AM