You are on page 1of 7

Eilfa - We Teach You Learn - solaris 10 tcp tuning

Page 1 of 7

solaris 10 tcp/ip tuning
In this solaris 10 tcp/ip tuning article i have collected the most essential and important tcp keys in order to achieve a better throughput, off course according to the situation being examined.

Please Note: When mentioned Temporary, it means that after reboot configuration will be erased while when mentioned Permanently, it means that after reboot the configurations will remain intact.

Fanout the incoming TCP/IP connections
Determines the number of squeues to be used to fanout the incoming TCP/IP connections. The incoming traffic is placed on one of the rings. If the ring is overloaded, packets are dropped. Default - 2 Range - 0 - nCPUs, where nCPUs is the maximum number of CPUs in the system Dynamic? - No. The interface should be plumbed again when changing this parameter. When to Change Consider setting this parameter to a value greater than 2 on systems that have 10 Gbps NICs and many CPUs. Use these commands to set temporarly and permanently the changes: Temporary - ndd -set /dev/tcp ip_soft_rings_cnt Permanently insert into /etc/system - set ip:ip_soft_rings_cnt=16

A value of 0 associates a new TCP/IP connection with the CPU that creates the connection. A value of 1 associates the connection with multiple squeues that belong to different CPUs. The number of squeues that are used to fanout the connection is based upon ip_soft_rings_cnt. Consider setting this parameter to 1 to spread the load across all CPUs in certain situations. For example, when the number of CPUs exceed the number of NICs, and one CPU is not capable of handling the network load of a single NIC, change this parameter to 1. Temporary - ndd -set /dev/tcp ip_squeue_fanout 1 Permanently insert into /etc/system - set ip:ip_squeue_fanout = 1

Remarks In Solaris, the available range of TCP/IP ports is 0 to 65535. However, there are some restrictions that apply: Ports in the range 0 to 1023 are reserved for privileged (root) services, such as telnetd, ftpd, and so on. Ports in the range 1024 to tcp_smallest_anon_port-1 are used for user

http://www.eilfa.com/solaris/solaris-10-tcp-tuning.html

12/02/13

html 12/02/13 . see tcp_slow_start_initial. and so on. Permanently insert into /etc/system . tcp_largest_anon_port: This is the largest TCP port number that may be used for anonymous connections. failures start to occur.eilfa.set ipc_tcp_conn_hash_size=8192 congestion window The maximum (Default 4)initial congestion window (cwnd) size in MSS of a TCP connection tcp_rexmit_interval_initial Temporary . TCP hash table size Check UNDER /etc/system: Controls the hash table size in the TCP module for all TCP connections (default 512). http://www. If your server is getting new client connections faster than it can bleed off sockets in the TIME_WAIT state. the following two ndd(1M) parameters can be used: tcp_smallest_anon_port: This determines the smallest TCP port number that may be used for an anonymous connection.We Teach You Learn . FONT server. This leaves the range 32768 to 65535 available for general TCP/IP connections. Decreasing the timeout increases the bleed-off rate.ndd -set /dev/tcp tcp_slow_start_initial 1 ndd -set /dev/tcp tcp_slow_start_after_idle 1 When to Change? For more information.Eilfa . Solaris allocates anonymous ports above 32768.solaris 10 tcp tuning Page 2 of 7 services such as NFS server daemon. Temporary . To limit the range of the port numbers allocated for the general use. When the list is full.ndd -set tcp:tcp_conn_hash_size=8192 Controls the hash table size in an IP module for all active (in ESTABLISHED state) TCP connections(default 512). the list will ultimately get full. The default value is 32768. The default value of this is 65535.com/solaris/solaris-10-tcp-tuning.ndd -set /dev/tcp tcp_time_wait_interval 3000 Setting this didnt give a better performance for a lighty web server. Default .60000 Temporary . kernel sockets The kernel keeps a list of sockets in the TIME_WAIT state.

insert into /etc/system Linux net.core.rmem_default = 16777216 Linux net.ipv4. TIME_WAIT ports These ensure that TIME_WAIT ports either get reused or closed fast.ipv4. If the initial cwnd size causes network congestion under special circumstances.tcp_synack_retries & net. decrease the value.tcp_fin_timeout = 1 in Solaris tcp_time_wait_interval Linux net.ipv4. but these are suitably high for our uses.nf_conntrack_max = 1048576 http://www.netdev_max_backlog = 262144 in Solaris to tcp_conn_req_max_q Linux tcp_slow_start_after_idle = 262144 SYN cookies protection Linux net.(The "syn cookies" violate the TCP spec thus solaris uses thier own mechanisem).com/solaris/solaris-10-tcp-tuning.html 12/02/13 .ipv4.rmem_max = 16777216 Linux net.ipv4.ip_conntrack_max = 1048576 net.ipv4.tcp_max_syn_backlog = 262144 in Solaris tcp_conn_req_max_q0 Linux net.eilfa.tcp_syn_retries = 2 in Solaris tcp_rexmit_interval_min 400 tcp_rexmit_interval_max 60000 tcp_ip_abort_interval 480000 tcp_rexmit_interval_initial 3000 conntrack You shouldn't be using conntrack on a heavily loaded server anyway. but Solaris has SYN flood protection enabled by default.core. insuring that if conntrack gets turned on.ipv4.netfilter.We Teach You Learn .Eilfa .tcp_syncookies = 1 . the box doesn't die net.tcp_tw_recycle = 1 TCP memory Linux net. Linux net.solaris 10 tcp tuning Page 3 of 7 ndd -set /dev/tcp tcp_slow_start_initial 2 ndd -set /dev/tcp tcp_slow_start_initial 1 When to Change? Do not change the value.ipv4.tcp_max_orphans = 262144 Linux net.core.

the netstat command shows that many of the sockets that are opened to the HTTP server are in the CLOSE_WAIT or FIN_WAIT_2 state.60000 Temporary . but CPU utilization stays high. Visible delays can occur for up to four minutes. http://www. Once enabled. When high connection rates occur. a keepalive probe packet is sent to the other end of the socket provided the connection has remained in the ESTABLISHED state and has been idle for the specified time frame. but CPU utilization stays high.com/solaris/solaris-10-tcp-tuning. Visible delays can occur for up to four minutes. Applications must enable this feature with the SO_KEEPALIVE socket option via the setsockopt(3SOCKET) socket call. This time frame is the value specified by the TCP tunable tcp_keepalive_interval. Default . during which time the server does not send any responses. as a way to clean up idle connections in situations like the ones mentioned above. After the applications complete the TCP/IP connection. Default .ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 67500 TCP keepalive TCP keepalive is a feature provided by many TCP implementations.675000 Temporary . the control blocks are kept for the specified time. a large backlog of the TCP/IP connections accumulates and can slow server performance. If the server stalls. The server can stall during certain peak periods. If the server stalls.html 12/02/13 . with all of the activity in system processes.Eilfa . a large backlog of TCP/IP connections accumulates and can slow server performance.solaris 10 tcp tuning Page 4 of 7 In Solaris use Dtrace script to track connections Dtrace script to track connections TCP/IP connection control blocks Notifies TCP/IP on how long to keep the connection control blocks closed. during which time the server does not send any responses. When high connection rates occur.ndd -set /dev/tcp tcp_time_wait_interval 3000 FIN_WAIT_2 state timer interval Specifies the timer interval prohibiting a connection in the FIN_WAIT_2 state to remain in that state.We Teach You Learn . with all of the activities in system processes. including Solaris. using the netstat command shows that many of the sockets opened to the HTTP server are in the CLOSE_WAIT or FIN_WAIT_2 state.eilfa. The server can stall during peak periods.

To limit the effects of SYN attacks we should enable some built-in protection mechanisms. as would be the case for any other unacknowledged packet. The first parameter tcp_conn_req_max_q controls the total number of full connections. Using the ndd command we can modify this value. Moreover. we can sometimes use techniques such as increasing the backlog queue size and minimizing the total time where a pending connection in kept in allocated memory (in the backlog queue). We know that every TCP port can handle a defined number of incoming requests. If no response is received by the time specified for the other TCP tunable. Hence the actual maximum idle time of a connection utilizing TCP keepalive.Eilfa . The second tcp_conn_req_max_q0 parameter defines how many halfopen connections are allowed without the dropping of incoming requests. tcp_ip_abort_interval. subsequent requests are silently dropped by the operating system. Keep in mind that TCP keepalive probes have no effect on inactive connections as long as the remote host is still responding to probes. In Sun Solaris 8.com/solaris/solaris-10-tcp-tuning. Additionally. host is probably under a SYN flooding attack. It will be retransmitted per the standard retransmission backoff algorithm.html 12/02/13 . As mentioned before. the source IP addresses of these incoming packets can be spoofed.solaris 10 tcp tuning Page 5 of 7 A keepalive probe packet is handled just like any other TCP packet which requires an acknowledgment (ACK) from the other end of the socket connection. However care should be taken to ensure the above parameters remain at a high enough value to avoid unnecessary traffic and other issues such as prematurely closing active connections in situations where a few packets have gone missing. which has no responding peer will therefore be: tcp_keepalive_interval + tcp_ip_abort_interval Default valuses respectively 7200000 480000 The above parameters are global and will affect the entire system. the default value is set to 1024. Run this command to count how many half-open connections are in the backlog queue at the moment netstat -s -P tcp | grep tcpHalfOpenDrop In Sun Solaris there are two parameters which control the maximum number of connections.ndd -set /dev/tcp tcp_keepalive_interval 300000 Backlog Queue The backlog queue is a large memory structure used to handle incoming packets with the SYN flag set until the moment the three-way handshake process is completed. When a maximum number of incoming connections is reached. the connection is terminated. An operating system allocates part of the system memory for every incoming connection. The backlog queue controls how many half-open connections can be handled by the operating system at the same time. when we detect a lot of connections in the SYN RECEIVED state.eilfa.We Teach You Learn . Temporary . http://www.

The only way to determine this empirically is to use ‘netstat –s | fgrep –i listendrop’.ndd -set /dev/tcp tcp_conn_req_max_q 128 OR 262144 Temporary .com/solaris/solaris-10-tcp-tuning. If tcpListenDrop is non-zero.html 12/02/13 . Temporary . #######Start of TCP/IP script############# http://www. establishing outgoing connections to destination servers such as replicas not responding quickly can cause long delays. If tcpListenDropQ0 is non-zero. increase tcp_conn_req_max_q0.ndd -set /dev/tcp tcp_ip_abort_interval 60000 (default is 480000) TCP/IP statistics These set of commands will present some of the TCP/IP statistics you will need in order to follow every change you make in your TCP/IP stack. if TCP has been retransmitting for tcp_ip_abort_interval period of time and it has not received any acknowledgment from the other endpoint during this period.We Teach You Learn .ndd -set /dev/tcp tcp_conn_req_max_q0 1024 OR 30000 Outgoing connection establishe time wait Some systems allow you to configure how long a system waits for an outgoing connection to be established.netstat -nP tcp |wc -l. Temporary .ndd -set /dev/tcp tcp_ip_abort_cinterval 10000 (default is 180000) Specifies the default total retransmission timeout value for a TCP connection.solaris 10 tcp tuning Page 6 of 7 It is pretty simple really: never change these parameters unless connections are refused because the values are too low.eilfa.netstat -s -P tcp | grep -E "tcpL" netstat -I bnx0 10 iostat -xn 10 TCP/IP script This script will assist you in configuring the TCP/IP parameters on your system. Temporary .Eilfa . netstat -nP tcp | grep WAIT | wc -l. When set too high. For a given TCP connection. TCP closes this connection. increase tcp_conn_req_max_q.

com/solaris/solaris-10-tcp-tuning.eilfa.We Teach You Learn .html 12/02/13 .solaris 10 tcp tuning Page 7 of 7 #!/sbin/sh ndd -set /dev/ip ip_forward_src_routed 0 #(Defalut value alreay set) ndd -set /dev/tcp tcp_rev_src_routes 0 #(Defalut value alreay set) ndd -set /dev/ip ip_forward_directed_broadcasts 0 #(Defalut value alreay set) *ndd -set /dev/tcp tcp_conn_req_max_q0 4096 #(Defalut value 1024) *ndd -set /dev/tcp tcp_conn_req_max_q 1024 #(Defalut value 128) ###Prevent System responding to ICMP timestamp requests ndd -set /dev/ip ip_respond_to_timestamp 0 #(Defalut value alreay set) ###Prevent System responding to ICMP timestamp Broadcast ndd -set /dev/ip ip_respond_to_timestamp_broadcast 0 #(Defalut value alreay set) ndd -set /dev/ip ip_respond_to_address_mask_broadcast 0 #(Defalut value alreay set) ndd -set /dev/ip ip_respond_to_echo_broadcast 0 ndd -set /dev/arp arp_cleanup_interval 60000 ndd -set /dev/ip ip_ire_arp_interval 60000 ndd -set /dev/ip ip_ignore_redirect 1 ndd -set /dev/ip ip_strict_dst_multihoming 1 ndd -set /dev/ip ip_send_redirects 0 ########END of TCP/IP script############### http://www.Eilfa .