Chapter -3

Kernel customization:

The kernel is a computer program that manages input/output requests from software, and
translates them into data processing instructions for the central processing unit and other
electronic components of a computer.

The kernel is a fundamental part of a modern computer's operating system. System services are
programs that load automatically either as part of an application's startup process or the
operating system startup process to support the different tasks required of t

he operating system.

A library is a collection of non-volatile resources used by computer programs, often to develop

software. These may include configuration data, documentation, help data, message templates,
prewritten code and subroutines, classes, values or type specifications. An application is a
program designed to perform a function or suite of related functions of benefit to an end user,
with examples being accounting, mathematical analysis, video editing, and word processing.

Kernel modules:

1
 o Kernel modules are pieces of code that can be loaded and unloaded into the kernel
upon demand. They extend the functionality of the kernel without the need to reboot
the system.
o To create a kernel module, you can read The Linux Kernel Module Programming Guide.
A module can be configured as built-in or loadable. To dynamically load or remove a
module, it has to be configured as a loadable module in the kernel configuration (the
line related to the module will therefore display the letter M).
o Ability to add functionality to operating system.
o Stored in lib/modules.
o Use the following command to list all drivers for various devices:
$ ls /lib/modules/$(uname -r)/kernel/drivers/

There are a few commands that allow you to manipulate the kernel. Each is quickly described
below, for more information say `man [command]`.
o load a module:

# modprobe module_name

To load a module by filename (i.e. one that is not installed in /lib/modules/$(uname -r)/):

# insmod filename [args]

To unload a module:

# modprobe -r module_name

Or, alternatively:

# rmmod module_name

For example: lp = line printer

root@:ls mod

root@:clear

root@:rmmod lp

root@:ls mod

root@:clear

root@:modprobe lp

2
Module path in kernel:

#cd /system/lib/modules

Modules# ls

4.2.0.16-generic

#cd 4.2.0.16-generic

4.2.0.16-generic#ls

Kernel

cd kernel

drivers,net etc

Need of kernel customization:

Traditionally, FreeBSD used a monolithic kernel. The kernel was one large program, supported a
fixed list of devices, and in order to change the kernel's behavior, one had to compile and then
reboot into a new kernel.

Today, most of the functionality in the FreeBSD kernel is contained in modules which can be
dynamically loaded and unloaded from the kernel as necessary. This allows the running kernel
to adapt immediately to new hardware and for new functionality to be brought into the kernel.
This is known as a modular kernel.

Occasionally, it is still necessary to perform static kernel configuration. Sometimes the needed
functionality is so tied to the kernel that it can not be made dynamically loadable. Some
security environments prevent the loading and unloading of kernel modules and require that
only needed functionality is statically compiled into the kernel.

Building a custom kernel is often a rite of passage for advanced BSD users. This process, while
time consuming, can provide benefits to the FreeBSD system. Unlike the GENERIC kernel, which
must support a wide range of hardware, a custom kernel can be stripped down to only provide
support for that computer's hardware. This has a number of benefits, such as:

o Faster boot time. Since the kernel will only probe the hardware on the system, the time
it takes the system to boot can decrease.

3
 o Lower memory usage. A custom kernel often uses less memory than
the GENERIC kernel by omitting unused features and device drivers. This is important
because the kernel code remains resident in physical memory at all times, preventing
that memory from being used by applications. For this reason, a custom kernel is useful
on a system with a small amount of RAM.
o Additional hardware support. A custom kernel can add support for devices which are
not present in the GENERIC kernel.
o Fix bug and upgrade system software
o To achieve performance.

Steps to upgrade /customize the kernel:

The Linux kernel is subject to more frequent revision than many other systems, owing to the
pace of its development. It must be recompiled when new changes are to be included, or when
an optimized kernel is required.

The compilation of a new kernel is a straightforward but time-consuming process. New kernel
distributions are obtained from any mirror of the Linux kernel site. First we back up the old
kernel, unpack the kernel sources into the operating system’s files and alias the kernel revision
to /usr/src/linux.

To check the running kernel:

$ cp /boot/

# Vmlinuz -4.2-0-16 generic

To check the present kernel running we have to use the following command

Uname –a

# 4.2.0-16 generic

The following steps are used to customize kernel are given below:

Step 1 : Download the latest kernel:

o Goto http://kernel.org/ website and get the latest version of the kernel source.
o we will will get 3.3.3 version using http://www.kernel.org/pub/linux/kernel/v3.0/linux-
3.3.3.tar.bz2 a
o Extract the source code using the tar command
tar xf linux-3.3.3.tar.bz2
cd linux-3.3.3

4
 o in linux-3.3.3 folder contain lots of files and folders. some folders are use following
purpose:
arch - this folder shows the what are the architecture port the Linux
crypto- contain cryptography implementations like AES, DES
drivers- all device drivers modules (IDE, SCSI, Ethernet device, Wireless)
fs- all file-system implementation code (ext4, FAT, NTFS)
net - all network protocol implementations (IPv4, IPv6, tcp, 802.11)

Step 2 : make the configuration file

While compile the kernel source code, we need configuration file. that configuration file
contain lots of variable to help to understand what are the modules we need to compile.

using make command we can build the configuration, but this command is interactive, its ask
more than 1000 question about enable or disable the particular module.

so the best way is copy the current Linux config file. its stored in /boot/config-<version>.
cp /boot/config-2.6.35.22-generic .config

now we got old configuration. now we change/add new configuration settings.

for this purpose therse is lots of options are avialable

make help ==>Provides the help

the following command are used to make configuration file:

#make menu config:

The command make config can now be used to set kernel parameters. More user-friendly
windows-based programs make xconfig or make menuconfig are also available, though the
former does require one to run X11 applications as root, which is a potential security faux pas.

Step 3 : Compile the Kernel:

the following command are used to make configuration file:

#sudo makes modules -install:

5
make modules_install
This step copy the all kernel modules (*.ko) to /lib/modules/<version>/kernel/ folder.

Install Kernel :

make install:
This step copy the the kenel from arch/x86/boot/bzImage to /boot folder and copy
the.config file to /boot/config-<latest-version> and generate the System.map file.

Create Initramfs file:

up to now kernel and its modules are compiled and installed. when next boot up time we
need to choose latest kernel. so we need to prepare the boot-loader and its support files. When
system turns on, after bios and boot loader load the kernel to main memory and mount initial
dummy file system as a root file system of system. this initial file system have necessary drivers
for disk hardware (SCSI or IDE) and mount the correct file system as a root file system.

so we need to create initramfs file using update-initramfs or mkinitfs tool

update-initramfs -c -k 3.3.3
here 3.3.3 is new kernel version.

System Performance Tuning :

System Performance Tuning answers one of the most fundamental questions you can ask about
your computer: How can I get it to do more work without buying more hardware? In the
current economic downturn, performance tuning takes on a new importance. It allows system
administrators to make the best use of existing systems and minimize the purchase of new
equipment. Well-tuned systems save money and time that would otherwise be wasted dealing
with slowdowns and errors. Performance tuning always involves compromises; unless system
administrators know what the compromises are, they can't make intelligent decisions. Tuning is
an essential skill for system administrators who face the problem of adapting the speed of a
computer system to the speed requirements imposed by the real world. It requires a detailed
understanding of the inner workings of the computer and its architecture. System Performance
Tuning covers two distinct areas: performance tuning, or the art of increasing performance for
a specific application, and capacity planning, or deciding what hardware best fulfills a given

6
role. Underpinning both subjects is the science of computer architecture. This book focuses on
the operating system, the underlying hardware, and their interactions.

Procedure 10-1. System Tuning Steps

1. Monitor system performance using various utilities.
2. Adjust specific values (for example, the maximum number of processes).
3. Reboot the system if necessary.
4. Test the performance of the new system to see if it is improved.
Note that performance tuning cannot expand the capabilities of a system beyond its hardware
capacity. You may need to add hardware, in particular another disk or additional memory, to
improve performance.

When is a fault not a fault? When it is an inefficiency. Sooner or later, user perception of system
performance passes a threshold. Beyond that threshold we deem the performance of a
computer to be unacceptably slow and we become irritated. Long before that happens, the
system itself recognizes the symptoms of a lack of resources and takes action to try to counter
the problem, but not always in the way we would like.

In order to identify a problem, we must first measure the performance. Again there are the two
issues: user perception of performance (interactive response time) and system throughput and
we have to choose the criterion we wish to meet. When the system is running slowly, it is
natural to look at what resources are being tested, i.e.

 What processes are running

 How much available memory the system has

 Whether disks are being used excessively

 Whether the network is being used heavily

 What software dependencies the system has (e.g. DNS, NFS).

1. Resource and Dependencies

2. Since all resources are scheduled by processes, it is natural to check the process table
first and then look at resource usage.

3. On Windows, one has the process manager and performance monitor for this. On Unix-
like systems, we check the process listing with ps aux, if a BSD compatible ps command
exists, or ps –ef if the system is derived from System V.

2. Hardware

7
Any effect which is introduced after the source will only reduce the performance in a chain of
data handling. A later component cannot ‘suck’ the data out of the source faster than the
source wants to deliver it. This tells us that the logical place to begin is with the system
hardware.

Corollary to principle (Performance).A system is limited by its slowest moving parts. Resources
with slowly moving parts, like disks, CD-ROMs and tapes, transfer data slowly and delay the
system. Resources which work purely with electronics, like RAM memory and CPU calculation,
are quick.

3. Software Tuning and Kernel Configuration

It is true that software is constrained by the hardware on which it runs, but it is equally true
that hardware can only follow the instructions it has received from software. If software asks
hardware to be inefficient, hardware will be inefficient. Software introduces many inefficiencies
of its own. Hardware and software tuning are inextricably intertwined. Software performance
tuning is a more complex problem than hardware performance tuning, simply because the
options we have for tuning software depend on what the software is, how it is written and
whether or not the designer made it easy for us to tune its performance. Some software
systems can be tuned individually. For instance, high availability server software such as WWW
servers and SMTP (E-mail) servers can be tuned to handle traffic optimally for heavy loads. We
can do the kernel configuration if we want to upgrade the System software so that hardware
can better communicate with system software.

4. Data Efficeiency

Efficiency of storage and transmission depends on the configuration parameters used to

manage disks and networks, and also on the amount of traffic the devices see.

Some file system formatting programs on Unix-like systems allow us to reserve certain
percentage of disk space for privileged users. If we have partitioned a host so as to separate
users from the operating system, then there is no need to reserve space on user disks. Better to
let users utilize the existing space until a real problem occurs. Tuning the network is a complex
subject and few operating systems allow us to do it at all. Solaris’ ndd command can be used to
configure TCP/IP parameters which can lead to noticeable performance improvements.

It’s really very tough job for every System or Network administrator to monitor and debug Linux
System Performance problems every day. These commands are available under all flavors
of Linux and can be useful to monitor and find the actual causes of performance problem. This
list of commands shown here are very enough for you to pick the one that is suitable for your
monitoring scenario

LINUX PERFORMANCE TUNING COMMAND :

8
Up time:

Uptime tells you how long the system has been running.

Description:

Uptime gives a one-line display of the following information:

o The current time

o How long the system has been running
o How many users are currently logged on
o The system load averages for the past 1, 5, and 15 minutes

mpstat:
The ability to accurately measure CPU usage of a system is essential to ensure optimal
operation of the applications or databases running on that system.
The mpstat Linux command is a utility that collects and displays information about CPU
utilization and performance statistics.

Top – Linux Process Monitoring:

Linux Top command is a performance monitoring program which is used frequently by many
system administrators to monitor Linux performance and it is available under
many Linux/Unix like operating systems. The top command used to dipslay all the running and
active real-time processes in ordered list and updates it regularly. It display CPU usage, Memory
usage, Swap Memory, Cache Size, Buffer Size, Process PID, User, Commands and much more. It
also shows high memory and cpu utilization of a running processess. The top command is much
userful for system administrator to monitor and take correct action when required. Let’s see
top command in action.

9
# top

VmStat – Virtual Memory Statistics:

o It is used to display all the running and active real time system.

o # vmstat
o procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
o r b swpd free inact active si so bi bo in cs us sy id wa st
o 1 0 0 810420 97380 70628 0 0 115 4 89 79 1 6 90 3 0

Lsof – List Open Files:

Lsof command used in many Linux/Unix like system that is used to display list of all the open
files and the processes. The open files included are disk files, network
sockets, pipes, devices and processes. One of the main reason for using this command is when
a disk cannot be unmounted and displays the error that files are being used or opened. With
this commmand you can easily identify which files are in use. The most common format for this
command is.

10
# lsof
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
init 1 root cwd DIR 104,2 4096 2/
init 1 root rtd DIR 104,2 4096 2/
init 1 root txt REG 104,2 38652 17710339 /sbin/init
init 1 root mem REG 104,2 129900 196453 /lib/ld-2.5.so
init 1 root mem REG 104,2 1693812 196454 /lib/libc-2.5.so
init 1 root mem REG 104,2 20668 196479 /lib/libdl-2.5.so
init 1 root mem REG 104,2 245376 196419 /lib/libsepol.so.1
init 1 root mem REG 104,2 93508 196431 /lib/libselinux.so.1
init 1 root 10u FIFO 0,17 953 /dev/initctl

Tcpdump – Network Packet Analyzer:

Tcpdump one of the most widely used command-line network packet analyzer or packets
sniffer program that is used capture or filter TCP/IP packets that received or transferred on a
specific interface over a network. It also provides a option to save captured packages in a file
for later analysis. tcpdump is almost available in all major Linux distributions.
# tcpdump -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
22:08:59.617628 IP tecmint.com.ssh > 115.113.134.3.static-mumbai.vsnl.net.in.28472: P
2532133365:2532133481(116) ack 3561562349 win 9648
22:09:07.653466 IP tecmint.com.ssh > 115.113.134.3.static-mumbai.vsnl.net.in.28472: P 116:232(116) ack 1 win
9648
22:08:59.617916 IP 115.113.134.3.static-mumbai.vsnl.net.in.28472 > tecmint.com.ssh: . ack 116 win 6

Netstat – Network Statistics:

Netstat is a command line tool for monitoring incoming and outgoing network packets
statistics as well as interface statistics. It is very useful tool for every system administrator to
monitor network performance and troubleshoot network related problems.

# netstat -a | more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:mysql *:* LISTEN
tcp 0 0 *:sunrpc *:* LISTEN
tcp 0 0 *:realm-rusd *:* LISTEN
tcp 0 0 *:ftp *:* LISTEN
tcp 0 0 localhost.localdomain:ipp *:* LISTEN
tcp 0 0 localhost.localdomain:smtp *:* LISTEN

11
tcp 0 0 localhost.localdomain:smtp localhost.localdomain:42709 TIME_WAIT
tcp 0 0 localhost.localdomain:smtp localhost.localdomain:42710 TIME_WAIT
tcp 0 0 *:http *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 *:https *:* LISTEN

Htop – Linux Process Monitoring:

Htop is a much advanced interactive and real time Linux process monitoring tool. This is much
similar to Linux top command but it has some rich features like user friendly interface to
manage process, shortcut keys, vertical and horizontal view of the processes and much more.
Htop is a third party tool and doesn’t included in Linux systems, you need to install it
using YUM package manager tool. For more information on installation read our article below.

#Htop

12
Iotop – Monitor Linux Disk I/O:

Iotop is also much similar to top command and Htop program, but it has
accounting function to monitor and display real time Disk I/O and processes.
This tool is much useful for finding the exact process and high used disk
read/writes of the processes.

# iotop

13
Iostat – Input/Output Statistics:

IoStat is simple tool that will collect and show system input and output storage device
statistics. This tool is often used to trace storage device performance issues
including devices, local disks, remote disks such as NFS.
# iostat
Linux 2.6.18-238.9.1.el5 (tecmint.com) 09/13/2012
avg-cpu: %user %nice %system %iowait %steal %idle
2.60 3.65 1.04 4.29 0.00 88.42
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
cciss/c0d0 17.79 545.80 256.52 855159769 401914750
cciss/c0d0p1 0.00 0.00 0.00 5459 3518
cciss/c0d0p2 16.45 533.97 245.18 836631746 384153384
cciss/c0d0p3 0.63 5.58 3.97 8737650 6215544
cciss/c0d0p4 0.00 0.00 0.00 8 0
cciss/c0d0p5 0.63 3.79 5.03 5936778 7882528

IPTraf – Real Time IP LAN Monitoring:

14
IPTraf is an open source console-based real time network (IP LAN) monitoring utility for Linux. It
collects a variety of information such as IP traffic monitor that passes over the network,
including TCP flag information, ICMP details, TCP/UDP traffic breakdowns, TCP connection
packet and byne counts. It also gathers information of general and detaled interface statistics of
TCP, UDP, IP, ICMP, non-IP, IP checksum errors, interface activity etc

Free:

o It display information about free and used memory in the system.

Managing heterogeneous system:

15
 o Heterogeneous computing refers to system that uses more than one kind of
processor.
For example CPU +GPU Gives heterogeneous system
o Heterogeneous system architecture utilizes multiprocessor type.

Almost all clusters expand after their initial installation. One of the reasons for this longevity of the
original hardware is the fact that processor clock speeds are not increasing as quickly as they had in the
past. Instead of faster clocks, processors now have multiple cores. Thus, many older nodes still deliver
acceptable performance (from a CPU clock perspective) and can provide useful work. Newer nodes
often have more cores per processor and more memory. In addition, many clusters now employ
accelerators from NVIDIA or AMD or coprocessors from Intel on some or all nodes. There may also be
different networks connecting different types of nodes (either 1/10 GbE or IB).

Another cluster trend is the use of memory aggregation tools such as ScaleMP. These tools allow
multiple cluster nodes to be combined into a large virtual Symmetric Multiprocessing (SMP) node. To
the end user, the combined nodes look like a large SMP node. Oftentimes, these aggregated systems are
allocated on traditional clusters for a period of time and released when the user has completed their
work, returning the nodes to the general node pool of the workload scheduler.

The combined spectrum of heterogeneous hardware and software often requires considerable expertise
for successful operation. Many of the standard open source tools do not make provision for managing or
monitoring these heterogeneous systems. Vendors often provide libraries and monitoring tools that
work with their specific accelerator platform, but make no effort to integrate these tools into any of the
open source tools. In order to monitor these new components, the administrator must “glue” the
vendor tools into whatever monitoring system they employ for the cluster, typically this means Ganglia
and Nagios. Although these tools provide hooks for custom tools, it is still incumbent upon the
administrator to create, test and maintain these tools for their environment.

Managing virtualized environments, like ScaleMP, can be similarly cumbersome to administrators.

Nodes will need the correct drivers and configuration before users can take advantage of the large
virtual SMP capabilities.

Heterogeneity may extend beyond a single cluster because many organizations have multiple clusters,
each with their own management environment. There may be some overlap of tools, however older
systems may be running older software and tools for compatibility reasons. This situation often creates
“custom administrators” that may have a steep learning curve when trying to assist or take over a new
cluster. Each cluster may have its own management style that further taxes administrators.

Ideally, cluster administrators would like the ability to have a “plug-and-play” environment for new
cluster hardware and software eliminating the need to weave new hardware into an existing monitoring
system or automate node aggregation tools with scripts. Bright Cluster Manager is the only solution that
offers this capability for clusters. All common cluster hardware is supported by a single, highly efficient
monitoring daemon. Data collected by the node daemons are captured in a single database, and

16
summarized according to the site’s preferences. There is no need to customize an existing monitoring
framework. New hardware and even new virtual ScaleMP nodes automatically show up in the familiar
management interface.

As shown in Figure 3, Bright Cluster Manager fully integrates NVIDIA GPUs and provides alerts and
actions for metrics such as GPU temperatures, GPU exclusivity modes, GPU fan speeds, system fan
speeds, PSU voltages and currents, system LED states, and GPU ECC statistics (Fermi GPUs only).
Similarly, Bright Cluster Manager includes everything needed to enable Intel Xeon Phi coprocessors in a
cluster using easy-to-install packages that provide the necessary drivers, SDK, flash utilities, and runtime
environment. Essential metrics are also provided as part of the monitoring interface.

Bright Cluster Manager manages accelerators and coprocessors used in hybrid-architecture systems for
HPC. Bright allows direct access to the NVIDIA GPU Boost technology through clock-speed settings.

Multiple heterogeneous clusters are also supported as part of Bright Cluster Manager. There is no need
to learn (or develop) a separate set of tools for each cluster in your organization.

17
Bright Computing’s integration with ScaleMP’s vSMP Foundation means that creating and dismantling a
virtual SMP node can be achieved with just a few clicks in the cluster management GUI. Virtual SMP
nodes can even be built and launched automatically on-the-fly using the scripting capabilities of the
cluster management shell. In a Bright cluster, virtual SMP nodes can be provisioned, monitored, used in
the workload management system, and have health checks running on them – just like any other node
in the cluster.

Recommendations for Managing Heterogeneity:

Make sure you can monitor all hardware in the cluster. Without the ability to monitor accelerators or
coprocessors, an unstable environment can be created for users that is difficult for administrators to
manage.

Heterogeneous hardware is now present in all clusters. Flexible software provisioning and monitoring
capabilities of all of components are essential. Try to avoid custom scripts that may only work for
specific versions of hardware. Many open source tools can be customized for specific hardware
scenarios thus minimizing the amount of custom work.

Make sure your management environment is the same across all clusters in your organization. A
heterogeneous management environment creates a dependency on specialized administrators and
reduces the ability to efficiently manage all clusters in the same way. Any management solution should
not interfere with HPC workloads.

Consider a versatile and robust tool like Bright Cluster Manager with its single comprehensive GUI and
CLI interface. All popular HPC cluster hardware and software technologies can be managed with the
same administration skills. The capability to smoothly manage heterogeneous hardware and software is
not available in any other management tool.

18