You are on page 1of 5

http://www.agilent.

com/comms/onenetworks

LAN Troubleshooting

Case in Point
Server Load Balancing and the MegaProxy Problem
To continue our efforts to help you with your network needs, we will be making several real-world network troubleshooting case studies available to you.
A brick and mortar retail firm that rolled-out their business online within the last year. The online retail business built customer facing web services that were hosted at a collocation facility. In order to scale the firm’s e-commerce capacity to the desired levels and to gain high availability of services, layer 4-7 server load balancing switches were placed in front of the web servers. Problem: Customer support was receiving calls from angry online shoppers with complaints of disconnections from the retail web site.

New case studies will be released every 2 to 3 weeks and will cover all topologies and a variety of network problems. Taken from Enterprise companies, Service Providers and Network Equipment Manufacturers, we will show you how companies, such as yours, have solved some of their network problems. You will have an opportunity to vote for the case studies of most interest to you. From these votes we will select the case studies for each posting. For additional case studies and to vote for the case studies you would like to see featured next, be sure to regularly visit our web site at:

http://www.agilent.com/comms/casestudies Feel free to pass this link on to your colleagues or friends.
1

http://www. There was no connection drops observed from the external client tests performed.agilent.com/comms/onenetworks LAN Troubleshooting Case Study: Server Load Balancing and the MegaProxy Problem A brick and mortar retail firm rolled-out their business online within the last year. In order to scale the firm’s e-commerce capacity to the desired levels and to gain high availability of services. The online retail business built customer facing web services that were hosted at a collocation facility. 2 . Case Study: Server Load Balancing and the MegaProxy Problem · An online retailer whose customer facing web services are hosted at a collocation facility · Server load balancing switches were placed in front of the web servers · Customer support is receiving calls from angry online shoppers with complaints of disconnections from the retail web site Customer support was receiving calls from angry online shoppers with complaints of disconnections from the retail web site. We began troubleshooting this issue by performing tests with client PCs that were placed on the Internet to make mock transactions to the retail web site. layer 4-7 server load balancing switches were placed in front of the web servers.

The captures confirmed a relatively moderate number of hard TCP session terminations and resets. SLB switches. Case Study: Server Load Balancing and the MegaProxy Problem · An analysis of the customers complaining was logged and analyzed · A new process assigned to customer service did identify a group of customers having the problem · A review of all settings on the SLB switches was done · Another internet based client test was performed after the examination of the customer data · The second attempt to test client connectivity from the internet confirmed the suspected ISP · During the mock transactions. the source IP address would change from time to time. The difference between a normal session and a purchase transaction session is that SSL will be invoked during the latter. suspected to be a pool of proxy servers An analysis of the customers complaining was logged and analyzed. and the public Internet with a multi-port protocol analyzer.com/comms/onenetworks LAN Troubleshooting Case Study: Server Load Balancing and the MegaProxy Problem · Tests were conducted where clients were placed externally out on the internet to make mock transactions to the web site · There were no connection drops from the external client tests performed · Packets were captured at the following locations: – Between the http servers and the SLB switches – External to the SLB switches · Packet captures confirmed a relatively moderate number of hard TCP session terminations · Limited debugging was turned on the SLB switches · The debug information from the SLB switches provided little added help to diagnose the issue Packets were captured between the HTTP servers. 3 . Limited debugging was turned on the SLB switches in a non-intrusive mode which eventually revealed little to help diagnose a cause. always rotating between the same few IP source address.agilent.http://www. Incorporating a special questionnaire into the customer service operations did result in the following two discoveries: 1) Nearly 90% of all customers experiencing the connection drops belonged to the same service provider of which was very popular and perhaps one of the largest. 2) Nearly all of the customers questioned had their session terminated while in the attempt to file a purchase.

the source IP address would change from time to time.agilent. It will treat the change as a completely new session. always rotating between the same few IP source addresses. In this instance.com/comms/onenetworks LAN Troubleshooting Case Study: Server Load Balancing and the MegaProxy Problem · SSL expects the source address not to change during a session · The SLB switches were configured to balance across the servers in a round-robin method · Each time a client/proxy changes it’s source IP address. Combined with switch debugging and packet captures. suspected to be a pool of proxy servers. a number of memberships to the suspect ISP were obtained and used.http://www. The sampling of clients used in the test had the same IP source address. 3) During the mock transactions. 4) SSL expects the source address not to change during a session and drops any connection when a source does change its IP address. This indicates that a proxy server might have been involved. the test confirmed the Internet based source of the problem as well as the following: 1) Patterns of source IP addresses from the clients were logged. This second attempt to test client connectivity from the Internet was more successful that time. 4 . the server load balancer treats that conversation as a new session which then round-robins it to the next server in line · The new server now receives the SSL session rejects the flow as it did not originate its establishment Another Internet based client test was performed after the examination of the customer data in item.

This creates a problem for SSL connections with the dynamic source IP assignments. It was later confirmed that this particular service provider did in fact use a rotation of proxy servers to add a level of security to its online community.agilent. the server load balancer treats that conversation as a new session which round-robins it to the next server in line. created what is now called the MegaProxy problem. the client is the proxy server. SSL sessions were dropped when the source IP of a client was changed. 5 .com/comms/onenetworks LAN Troubleshooting Case Study: Server Load Balancing and the MegaProxy Problem Root Cause: · It was confirmed that this particular service provider did use a pool of proxy servers for securing its community · SSL sessions are dropped when the source IP of a client is changed Solution: · In order to provide persistence in session connection between a dynamic source and a secure web server. But by doing so. The new server now receives the SSL session rejects the flow as it did not originate its establishment. the SLB switch was outfitted with code that could passively capture the SSL session ID. In order to provide persistence in session connection between a dynamic source and a secure web server. The SSL session ID then becomes the unique identifier and when the source IP changes in the flow. Each time a client/proxy changes it’s source IP address. From the http server’s perspective. The SLB switches were configured to balance across the servers in a round-robin method. the SLB switch overrides it’s server load balancing process to maintain persistence. the SLB switch was configured to capture the SSL session ID · The session ID is used to maintain the connection between client and server regardless of what IP address changes may occur · The new configuration overrides it’s server load balancing process to maintain persistence Review of all settings on the SLB switches.http://www.