You are on page 1of 28

Seminar Report 2009-10

Linux Virtual Server

Linux Virtual Server (LVS) is an open source technology which can be used to construct a scalable and highly available server using a collection of real servers. The Linux Virtual Server Project (LVS) allows load balancing of networked services such as web and mail servers using Layer 4 Switching. It is extremely fast and allows such services to be scaled to service 10s or 100s of thousands of simultaneous connections. Here we describes about various features of LVS to load balance Internet services, and how this can be made highly available using tools such as such as heartbeat and keep alived. It will also cover more advanced topics which have been the subject of recent development including maintaining active connections in a highly available environment and using active feedback to better distribute load.

Delta Computer Education, Kalpetta



Seminar Report 2009-10

Linux Virtual Server

Virtual server is a highly scalable and highly available server built on a cluster of real servers. The architecture of server cluster is fully transparent to end users, and the users interact with the cluster system as if it were only a single high-performance virtual server. The real servers and the load balancers may be interconnected by either high-speed LAN or by geographically dispersed WAN. The load balancers can dispatch requests to the different servers and make parallel services of the cluster to appear as a virtual service on a single IP address, and request dispatching can use IP load balancing technologies or application-level load balancing technologies. Scalability of the system is achieved by transparently adding or removing nodes in the cluster. High availability is provided by detecting node or daemon failures and reconfiguring the system appropriately. The Linux Virtual Server (LVS) implements layer 4 switching in the Linux Kernel. This allows TCP and UDP sessions to to be load balanced between multiple real servers. Thus it provides a way to scale Internet services beyond a single host. HTTP and HTTPS traffic for the World Wide Web is probably the most common use. Though it

Delta Computer Education, Kalpetta



LVS can be used. It is easily able to load balance a saturated 100Mbitethernet link using inexpensive commodity hardware. Delta Computer Education. however it is able to load balance connections from end users running any operating system to real servers running any operating system.000 simultaneous connections. It is also able to load balance saturated 1Gbit link and beyond using higher-end commodity hardware. As long as the connections use TCP or UDP. from email to the X Windows System. LVS is very high performance. LVS itself runs on Linux.Seminar Report 2009-10 Linux Virtual Server can also be used for more or less any service. It is able to handle upwards of 100. Kalpetta DHNE 3 .

mail or media service. reliability and service ability. The LVS cluster system is also known as load balancing server cluster. To check the integrity of the services on each real server. Goals The basic goal of the Linux Virtual Server is to build a highperformance and highly available server for Linux using clustering technology. The backup LVS router monitors the active LVS router and takes over from it in case the active LVS router fails. such as a scalable web. which provides good scalability. Delta Computer Education.Seminar Report 2009-10 Linux Virtual Server Applications The Linux Virtual Server can be used to build highly scalable and high available network services. The active LVS router serves two roles: • • To balance the load across the real servers. Kalpetta DHNE 4 .

and so on. The load balancer directs network connections from clients who know a single IP address for services. • Backend storage. is the front end to the service as seen by the outside world. such as web. Kalpetta DHNE 5 . consits of a cluster of servers that implement the actual services. Figure: The 3-tier Architecture of Linux Virtual Server The load balancer handles incoming connections using IP load balancing Delta Computer Education.Seminar Report 2009-10 Linux Virtual Server SYSTEM ARCHITECTURE The three-tier architecture of LVS illustrated in Figure includes: • Load balancer. to a set of servers that actually perform the work. ftp. Server pool. mail. so that it is easy for servers to keep the same content and provide the same services. provides the shared storage for the servers. • . dns.

the aggregate performance should scale linearly with the number of nodes in the system. Since the commodity servers are used as building blocks. The server nodes in the above architecture may be replicated for either scalability or high availablity. Kalpetta DHNE 6 . it selects servers from the server pool. maintains the state of concurrent connections and forwards packets. thus a load balancer can schedule a large number of servers and it will not be a bottleneck of the whole system soon.Seminar Report 2009-10 Linux Virtual Server techniques. so that the handling overhead of the load balancer is low. more server nodes can be added to handle the increasing workload. Since the dependence of most network services is often not high. When the load to the system saturates the capacity of existing server nodes. Scalability is achieved by transparently adding or removing a node in the cluster. One of the advantages of a clustered system is that it has hardware and software redundancy. Therefore. before the load balancer becomes a new bottleneck of the system. and all the work is performed inside the kernel. the performance/ cost ratio of the whole systemis as high as that of commodity servers. High availability can be provided by detecting node or daemon failures and reconfiguring the system Delta Computer Education. the load balancer can handle much larger number of connections than a general server.

Seminar Report 2009-10 Linux Virtual Server appropriately so that the workload can be taken over by the remaining nodes in the cluster. In order to prevent the failure of the load balancer. and the primary will take over the virtual IP Delta Computer Education. When the primary recovers from its failure. the load balancer may become a single failure point of the whole system. Now. We usually have cluster monitor daemons running on the load balancer to monitor the health of server nodes. When the heartbeat daemon on the backup cannot hear the health message from the primary in the specified time. the monitor will remove or disable the server in the scheduling table of the load balancer. they heartbeat the health message through heartbeat channels such as serial line and UDP periodically. there are two methods. it will use ARP spoofing (gratutious ARP) to take over the virtual IP address to provide the load-balancing service. if a server node cannot be reached by ICMP ping or there is no response of the service in the specified period. Kalpetta DHNE 7 . Two heartbeat daemons run on the primary and the backup. we need setup a backup of the load balancer. One is that the primary becomes to the backup of the functioning load balancer. so that the load balancer will not schedule new connections to the failed one and the failure of server nodes can be masked. the other is that the daemon receives the health message from the primary and releases the virtual IP address.

Kalpetta DHNE 8 . we patch the Linux kernel (2. However. to a set of servers that actually perform the work. the failover or the takeover of the primary will cause the established connection in the state table lost in the current implementation. no matter it talks TCP or UDP protocol. shared on a network file system. The box running Linux Virtual Server act as a load balancer of network connections from clients who know a single IP address for a service. In general. We call data communication between a client’s socket and a server’s socket connection.0 and 2. which will require the clients to send their requests again. The contents are either replicated on each server’s local disk.Seminar Report 2009-10 Linux Virtual Server address. The following subsections describe the working principles of three techniques and their advantages and disadvantages. • Linux Virtual Server via NAT Delta Computer Education. real servers are idential. or served by a distributed file system. IP Load Balancing Techniques Since the IP load balancing techniques have good scalability. they run the same service and they have the same set of contents.2) to support three IP load balancing techniques. LVS/NAT. LVS/TUN and LVS/DR.

Figure: Architecture of LVS/NAT Delta Computer Education.Seminar Report 2009-10 Linux Virtual Server Due to the shortage of IP address in IPv4 and some security reasons. This feature can be used to build a virtual server. Network address translation relies on the fact that the headers of packets can be adjusted appropriately so that clients believe they are contacting one IP address.e. but servers at different IP addresses believe they are contacted directly by the clients. Kalpetta DHNE 9 . parallel services at the different IP addresses can appear as a virtual service on a single IP address. The need for network address translation arises when hosts in internal networks want to access or to be accessed on the Internet. i. more and more networks use private IP addresses which cannot be used on the Internet.

a request packet destined for virtual IP address (the IP address to accept requests for virtual service) arrives at the load balancer. When a connection terminates or timeouts. the load balancer rewrites the source address and port of the packets to those of the virtual service. Kalpetta DHNE 10 . a real server is selected from the cluster by a scheduling algorithm. and the packet is forwarded to the server. the destination address and the port of the packet are rewritten to those of the selected server. if they are matched for a virtual service according to the virtual server rule table. The load balancer examines the packet’s destination address and port number. and the connection is added into the hash table which records connections. the connection record will be removed in the hash table. The workflow of LVS/NAT is as follows: When a user accesses a virtual service provided by the server cluster. the connection can be found in the hash table and the packet will be rewritten and forwarded to the right server. • Linux Virtual Server via IP Tunneling Delta Computer Education. When response packets come back.Seminar Report 2009-10 Linux Virtual Server The architecture of Linux Virtual Server via NAT is illustrated in Figure. The load balancer and real servers are interconnected by a switch or a hub. Then. When an incoming packet belongs to an established connection.

thus the service can still appear as a virtual service on a single IP address. The architecture of LVS/DR is illustrated in Figure 4. The virtual IP address is shared by real servers and the load balancer. and the load balancer has an interface configured with the virtual IP address to accept incoming packets. dvantages and Disadvantages The characteristics of three IP load balancing techniques Delta Computer Education. The load balancer and the real servers must have one of their interfaces physically linked by an uninterrupted segment of LAN such as a HUB/Switch. and the servers process the requests and return the results to the clients directly. Kalpetta DHNE 11 . All real servers have their loopback alias interface configured with the virtual IP address. which allows datagrams destined for one IP address to be wrapped and redirected to another IP address.Seminar Report 2009-10 Linux Virtual Server IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagram. • Linux Virtual Server via Direct Routing This IP load balancing approach is similar to the one implemented in IBM’s NetDispatcher. This technique can be used to build a virtual server that the load balancer tunnels the request packets to the different servers.

real servers can run any operating system that supports TCP/IP protocol. a LVS/TUN load balancer may schedule over 100 general real servers and it won’t be the bottleneck of the system. The load balancer can schedule 15 servers if the average throughout of real servers is 600KBytes/s.Seminar Report 2009-10 Linux Virtual Server are summarized in Table 1. because both request and response packets need to be rewritten by the load balancer. The load balancer may be a bottleneck of the whole system when the number of server nodes increase to around 20 which depends on the throughout of servers. The disadvantage is that the scalability of LVS/NAT is limited. Supposing the average length of TCP packets is 536 Bytes and the average delay of rewriting a packet is around 60us on the Pentium processor (this can be reduced a little by using of faster processor). the maximum throughout of the load balancer is 8.  Linux Virtual Server via NAT In LVS/NAT. Kalpetta DHNE 12 .  Linux Virtual Server via IP tunneling For most Internet services (such as web service) that request packets are often short and response packets usually carry large amount of data. Delta Computer Education.93 Mbytes/s. and only one IP address is needed for the load balancer and private IP addresses can be used for real servers.

LVS/DR doesn’t have tunneling overhead .Seminar Report 2009-10 Linux Virtual Server because the load balancer just directs requests to the servers and the servers reply the clients directly. extremely good to build a virtual proxy server because when the proxy servers receive requests. a LVS/DR load balancer processes only the client-to-server half of a connection. Compared to LVS/TUN. Delta Computer Education. the load balancer and each server must be directly connected to one another by a single uninterrupted segment of a local-area network. LVS/TUN requires servers support IP Tunneling protocol. LVS/TUN should be applicable to servers running other operating systems. LVS/TUN can be used to build a virtual server that takes huge load. Since the IP tunneling protocol is becoming a standard of all operating systems. This can greatly increase the scalability of virtual server. However. Therefore. Kalpetta DHNE 13 . This feature has been tested with servers running Linux. but it requires the server OS has loopback alias interface that doesn’t do ARP response. and the response packets can follow separate network routes to the clients.  Linux Virtual Server via Direct Routing Like LVS/TUN. LVS/TUN has good scalability. they can access the Internet directly to fetch objects and return them to the clients directly.

and read the virtual server rules through /proc file system. It looks up the “VS Rules” hash table for new connections. Figure: Implementation of LVS The connection hash table is designed to hold millions of concurrent connections. The “IPVSADM” user-space program is to administrator virtual servers. The “VS Schedule&Control Module” is the main module of LVS. it hooks two places at IP packet traversing inside kernel in order to grab/rewrite IP packets to support IP load balancing. it uses setsockopt function to modify the virtual server rules inside the kernel. and checks the “Connection Hash Table” for established connections. Kalpetta DHNE 14 . and each connection entry only occupies 128 Delta Computer Education.Seminar Report 2009-10 Linux Virtual Server IMPLEMENTATION ISSUES The system implementation of Linux Virtual Server is illustrated in Figure.

and some geographically distributed servers through LVS/TUN. The incoming ICMP packets for virtual services will be forwarded to the right real servers. for example. address. This is important for error and control notification between clients and servers. Delta Computer Education. and they can also be used together in a single cluster. and outgoing ICMP packets from virtual services will be altered and sent out correctly. packets are forwarded to some servers through LVS/NAT method. a load balancer of 256 Mbytes free memory can have two million concurrent connections. Slow timer is ticked every second to collect stale connections. LVS implements three IP load balancing techniques. LVS implements ICMP handling for virtual services.Seminar Report 2009-10 Linux Virtual Server bytes effective memory in the load balancer. The hash table size can be adapted by users according to their applications. and the client <protocol. For example. They can be used for different kinds of server clusters. such as the patch MTU discovery. Kalpetta DHNE 15 . port>is used as hash key so that hash collision is very low. some servers through LVS/DR.

The applet makes requests to the cluster of servers to collect load information of all the servers. we can see that there are many ways to dispatch requests to a cluster of servers in the different levels. one end is the client. the other end is the server. they will potentially increase network traffic by extra querying or probing. then chooses a server based on that information and forwards requests to that server. The server-side Round-Robin DNS approach The NCSA scalable web server is the first prototype of a scalable web server using the Round-Robin DNS approach. However. The applet tries other servers when it finds the chosen server is down. these client-side approaches are not clienttransparent. Existing request dispatching techniques can be classified into the following categories: The client-side approach Berkeley’s Smart Client suggests that the service provide an applet running at the client side. Moreover. Kalpetta DHNE 16 . The RRDNS server maps a single name to the different IP addresses in a roundrobin manner so that the different clients will access the different Delta Computer Education.Seminar Report 2009-10 Linux Virtual Server ALTERNATIVE APPROACHES In the client/server applications. they requires modification of client applications. and there may be a proxy in the middle. Based on this scenario. so they cannot be applied to all TCP/IP services.

it is not so reliable. and with high values the dynamic load imbalance will get even worse. the scheduling granularity is per host. thus it is not easy for a server to handle its peak load. and finally return them to the clients. when a server node fails. then get the results. Even the TTL value is set with zero. due to the caching nature of clients and hierarchical DNS system. and others may just surf a few pages and leave. because some clients (such as a proxy server) may pull lots of pages from the site. this approach requires to establish two TCP connections for each request. The server-side application-level scheduling approach EDDIE. Futhermore. They all forward HTTP requests to different web servers in the cluster. Reverse-proxy and SWEB use the application-level scheduling approach to build a scalable web server. the clients who maps the name to the IP address will find the server is down. one is Delta Computer Education. Kalpetta DHNE 17 . However. However.Seminar Report 2009-10 Linux Virtual Server servers in the cluster for the ideal situation and load is distributed among the servers. it easily leads to dynamic load imbalance among the servers. The TTL(Time To Live) value of a name mapping cannot be well chosen at RR-DNS. and the problem still exists even if they press ”reload” or ”refresh” button in the browsers. with small values the RR-DNS will be a bottleneck. different client access pattern may lead to dynamic load imbalance.

directly forwards packets to servers that is configured with router Delta Computer Education. the MagicRouter doesn’t survive to be a useful system for others. Thus the application-level load balancer will be a new bottleneck soon when the number of server nodes increases. The TCP router changes the destination address of the request packets and forwards the chosen server. NetDispatcher. IBM’s TCP router uses the modified Network Address Translation approach to build scalable web server on IBM scalable Parallel SP-2 system.Seminar Report 2009-10 Linux Virtual Server between the client and the load balancer. The server-side IP-level scheduling approaches Berkeley’s MagicRouter and Cisco’s LocalDirector use the Network Address Translation approach similar to the NAT approach used in Linux Virtual Server. the other is between the load balancer and the server. The advantage of the modified approach is that the TCP router avoids rewriting of the reply packets. the successor of TCP router. Kalpetta DHNE 18 . the delay is high. The overhead of dealing HTTP requests and replies in the application-level is high. that server is modified to put the TCP router address instead of its own address as the source address in the reply packets. and they only support part of TCP protocol. However. the LocalDirector is too expensive. the disadvantage is that it requires modification of the kernel code of every server in the cluster.

Two dispatching techniques are used. The advantage is that the rewriting of response packets can be avoided. and the local filtering also requires modification of the kernel code of server.Seminar Report 2009-10 Linux Virtual Server address on non arp-exported interfaces. but NetDispatcher is a very expensive commercial product. The approach. the other is based on packet broadcasting and local filtering. On the backup LVS router. The disadvantage is that it cannot be applied to all operating systems because some operating systems will shutdown the network interface when detecting IP address collision. Kalpetta DHNE 19 . pulse sends a heartbeat to the Delta Computer Education. similar to the LVS/DR in Linux Virtual Server. ONE-IP requires that all servers have their own IP addresses in a network and they are all configured with the same router address on the IP alias interfaces. one is based on a central dispatcher routing IP packets to different servers. has good scalability. Components of a Running LVS Cluster The pulse daemon runs on both the active and passive LVS routers.

the lvs daemon instructs ipvsadm to remove that real server from the IPVS routing table. sends a command to the active LVS router via both the public and private network interfaces to shut down the lvs daemon on the active LVS router. it initiates failover by calling send_arp to reassign all virtual IP addresses to the NIC hardware addresses (MAC address) of the backup LVS router. and starts the lvs daemon on the backup LVS router to accept requests for the configured virtual servers. Delta Computer Education. Each nanny process checks the state of one configured service on one real server. On the active LVS router. the LVS daemon calls the ipvsadm utility to configure and maintain the IPVS (IP Virtual Server) routing table in the kernel and starts a nanny process for each configured virtual server on each real server. pulse starts the lvs daemon and responds to heartbeat queries from the backup LVS router. Kalpetta DHNE 20 . To an outside user accessing a hosted service (such as a website or database application).Seminar Report 2009-10 Linux Virtual Server public interface of the active router to make sure the active LVS router is properly functioning. and tells the lvs daemon if the service on that real server is malfunctioning. Once started. LVS appears as one server. If a malfunction is detected. If the backup LVS router does not receive a response from the active LVS router.

The first option is preferred for servers that do not allow large numbers of users to upload or change data on the real servers. such as an ecommerce website.Seminar Report 2009-10 Linux Virtual Server However. The LVS-router tier consists of one active LVS router and one backup LVS router. which in Delta Computer Education. Each LVS router has two network interfaces: one connected to a public network (Internet) and one connected to a private network. Kalpetta DHNE 21 . the active LVS router uses Network Address Translation (NAT) to direct traffic from the public network to real servers on the private network. adding a third layer is preferable. If the real servers allow large numbers of users to modify data. Add a third layer to the topology for shared data access. Two-Tier LVS Topology The figure below shows a simple LVS configuration consisting of two tiers: LVS routers and real servers. the user is actually accessing real servers behind the LVS routers. In this figure. Because there is no built-in component in LVS to share the data among real servers. The real-server tier consists of real servers connected to the private network. A network interface connected to each network allows the LVS routers to regulate traffic between clients on the public network and the real servers on the private network. you have have two basic options: • • Synchronize the data across the real servers.

For instance. and which is assigned to one or more virtual servers. This is a publicly-routable address that the administrator of the site associates with a fully-qualified domain name. The real servers pass all public traffic through the active LVS each virtual server can be associated with a Delta Computer Education. also known as floating IP addresses. Two-TierLVS Topology Service requests arriving at an LVS router are addressed to a virtual IP address or VIP. From the perspective of clients on the public network. VIP addresses may be aliased to the same device that connects the LVS router to the public network.Seminar Report 2009-10 Linux Virtual Server turn provide services as requested. the LVS router appears as one entity. Alternatively.example. thus maintaining a presence at that IP address. if eth0 is connected to the Internet. then multiple virtual servers can be aliased to eth0:1. such as www. Kalpetta DHNE 22 . Note that a VIP address migrates from one LVS router to the other during a failover.

all the real servers are treated as equals without regard to capacity or load. Using this algorithm. and FTP traffic can be handled on eth0:2. the active LVS router routes the requests from the public network (Internet) to the second tier — real servers.Seminar Report 2009-10 Linux Virtual Server separate device per service. Three-Tier LVS Topology The figure below shows a typical three-tier LVS configuration. • Weighted Round-Robin Scheduling — Distributes each request sequentially around a pool of real servers but gives more jobs to servers with greater capacity. Each real server Delta Computer Education. This is a preferred choice if there are significant differences in the capacity of real servers in a server pool. Capacity is indicated by a userassigned weight factor. For example. In the example. The redirection is based on one of eight load-balancing algorithms: • Round-Robin Scheduling — Distributes each request sequentially around a pool of real servers. The role of the active LVS router is to redirect service requests from virtual IP addresses to the real servers.Only one LVS router is active at a time. HTTP traffic can be handled on eth0:1. which is then adjusted up or down by dynamic load information. Kalpetta DHNE 23 .

highly available server and accessed by each real server via an exported NFS directory or Samba share. Kalpetta DHNE 24 . Delta Computer Education. This topology is also recommended for websites that access a central. high-availability database for transactions.Seminar Report 2009-10 Linux Virtual Server then accesses a shared data source of a Red Hat cluster in the third tier over the private network. where accessible data is stored on a central. Three-TierLVS Topology This topology is suited well for busy FTP servers.

Kalpetta DHNE 25 .Seminar Report 2009-10 Linux Virtual Server FORWARDING PACKETS The Linux Virtual Server has three different ways of forwarding packets. The IP packet is not modified. The most common use of this is IP masquerading which is often used to enable RFC 1918[2] private networks to access the Internet. • Network Address Translation (NAT): A method of manipulating the source and/or destination port and/or address of a packet. packets are received from end users and the destination port and IP address are changed to that of the chosen real server. IP-IP encapsulation (tunnelling) and direct routing. The real server may send Delta Computer Education. so the real servers must be configured to accept traffic for the virtual server's IP address. Return packets pass through the linux director at which time the mapping is undone so the end user sees replies from the expected source. In the context of layer 4 switching. • Direct Routing: Packets from end users are forwarded directly to the real server. This can be done using a dummy interface or packet filtering to redirect traffic addressed to the virtual server's IP address to a local port. network address translation (NAT).

Thus. the linux director does not need to be in the return path. The main advantage of using tunnelling is that real servers can be on a different networks. Kalpetta DHNE 26 . • IP-IP Encapsulation (Tunnelling): Allows packets addressed to an IP address to be redirected to another address. rather than just manipulating the ethernet frame. except that when packets are forwarded they are encapsulated in an IP packet.Seminar Report 2009-10 Linux Virtual Server replies directly back to the end user. Delta Computer Education. In the context of layer 4 switching the behaviouris very similar to that of direct routing. possibly on a different network.

LVS itself is a very powerful tool and has many features that were not within the scope of this presentation. firewall marks to group virtual services. Tools such as heartbeat. Beyond that there is much scope for further expanding the functionality of LVS to meet the new needs of users and to reflect the ever increasing complexity of the Internet. specialised scheduling algorithms and various tuning parameters. As well as connection synchronisation and active techniques to multiple linux directors to better work together. Delta Computer Education.Seminar Report 2009-10 Linux Virtual Server CONCLUSION LVS is an effective way to implement clustering of Internet services. ldirectord and keepalived can be used to give the cluster high availability. These include. There are a number of other techniques that can be used to further enhance LVS clusters including using active feedback to determine the proportion of connections allocated to each of the real servers. Kalpetta DHNE 27 .

ultramonkey.linuxvirtualserver.redhat. 2.htm l Delta Computer Report 2009-10 Linux Virtual Server REFERENCE 1. 3. http://www.html http://en. Kalpetta DHNE 28 .