Essentials for Scientific Computing:

Source Code, Compilation and Libraries

Day 9
Ershaad Ahamed
TUE-CMS, JNCASR

May 2012

1 Using GNU Make

We’ve seen earlier that programs can consist of many source files. As projects
become larger, the number of source files can quickly become very large. There
might also be dependencies between source files. During the development of
such projects, changes may be made to source files in order to add features or
fix bugs. A change in a source file requires the object files to be recompiled
and relinked in order to create a new executable. A change in a header file will
affect all source files that include it and thus requires that their corresponding
object files be recompiled and relinked. This task of compiling and recompiling
sources when changes are made can quickly become tedious and error-prone as
the number of source files increase and dependencies become more complex.
Add to that the possibility of compiling for different hardware and operating
system versions and platforms, and it become quite impossible for a programmer
to track what needs to be recompiled by hand. Although the entire project can
be recompiled whenever a change is made, this can mean the difference between
wasting an hour when only a few seconds worth of recompilation and relinking
is necessary.
The make program is intended to automate this task. There exist a few
variants of make, each with their own extensions and features added to the
historical UNIX make. The most popular version on Linux systems is GNU
Make which we will be using.
In order to use make, we need to specify dependencies between files and the
commands that need to be run in order to rebuild or recompile them. make uses
this information in addition to the timestamps of files (which tells it whether a
file has been modified) to determine which parts of the source need to be recom-
piled and relinked in order to rebuild the executable. make reads dependency
information from a ‘makefile’ which is usually named Makefile or makefile,
and is written in a syntax which is described below.
Makefiles consists of rules that specify targets, prerequisites and commands
and look like.
target: prereq1 prereq2
commands

1
It is important to note that the space at the start of the commands
line must be a tab character, otherwise it wont work. What this tells
make is that target depends on the files prereq1 and prereq2. The commands
tell make how to build target from the prerequisites.
Using this information let us write a rule for our example program above.
test_link: test_link.c libmymath.so
gcc -o test_link test_link.c libmymath.so

This rule tells make that the target file test link, which is the executable we
eventually want to build depends on test link.c and libmyshared.so. When-
ever one of these files changes, test link needs to be rebuilt, and this is done
with the command on the next line, gcc -o test_link test_link.c libmymath.so}.
Remember that a tab character precedes the line with the gcc command.
Now that we have a rule in place in our makefile called Makefile, we can
run the make command. By default, running make will cause the first target (if
there are multiple rules) to be built. Typing make at the command line.
make
Gives the output.

make: *** No rule to make target ‘libmymath.so’,

needed by ‘test_link’. Stop.
This is an error and tells us that make did not find the file libmymath.so. And
since there is no rule telling make how to build the target libmymath.so, it
does not know what commands need to be run build it. Let’s add a rule for
the target libmymath.so and since we know that libmymath.so depends on
libcubed.o and libpowerfour.o we also add rules for building libcubed.o
and libpowerfour.o from their sources.

2
test_link: test_link.c libmymath.so
gcc -o test_link test_link.c libmymath.so

libmymath.so: libcubed.o libpowerfour.o

gcc -shared -Wl,-soname,libmymath.so -o libmymath.so \
libcubed.o libpowerfour.o

libcubed.o: libcubed.c
gcc -fPIC -c libcubed.c

libpowerfour.o: libpowerfour.c
gcc -fPIC -c libpowerfour.c

We use line continuation for lines that are too long. Running make gives us the
output.
gcc -fPIC -c libcubed.c
gcc -fPIC -c libpowerfour.c
gcc -shared -Wl,-soname,libmymath.so -o libmymath.so \
libcubed.o libpowerfour.o
gcc -o test_link test_link.c libmymath.so
Make has successfully built test link from the sources using the correct se-
quence of steps to satisfy dependencies. If we run make once again, we get the
output.
make: ‘test_link’ is up to date.
Make did not detect any change in prerequisite files for test link and did the
right thing by not recompiling the entire source.
Let us edit one of the source files, say, libcubed.c and save it. It doesn’t
really matter whether any real change is made to the file, since make looks at
the time when the file was last modified and the modification time is updated
when we save the file. Running make, we get.
gcc -fPIC -c libcubed.c
gcc -shared -Wl,-soname,libmymath.so -o libmymath.so \
libcubed.o libpowerfour.o
gcc -o test_link test_link.c libmymath.so
Notice that make only recompiled libcubed.c into an object file and not libpowerfour.c.
It also rebuilt libmymath.so and test link since they depend on libcubed.o.
It is in this way that make analyses dependencies and avoids unnecessary build
steps.

1.1 Pattern Rules and Automatic Variables

1.1.1 Pattern Rules
Suppose that, as our project grows, we add more functions into our dynamic li-
brary libmymath.so. Imagine we add the functions powerfive() and squareroot(),
with each being in a separate source file. Our new makefile will look like.

3
test_link: test_link.c libmymath.so
gcc -o test_link test_link.c libmymath.so

libmymath.so: libcubed.o libpowerfour.o libpowerfive.o libsquareroot.o

gcc -shared -Wl,-soname,libmymath.so -o libmymath.so \
libcubed.o libpowerfour.o libpowerfive.o libsquareroot.o

libcubed.o: libcubed.c
gcc -fPIC -c libcubed.c

libpowerfour.o: libpowerfour.c
gcc -fPIC -c libpowerfour.c

libpowerfive.o: libpowerfive.c
gcc -fPIC -c libpowerfive.c

libsquareroot.o: libsquareroot.c
gcc -fPIC -c libsquareroot.c
It’s clear that our makefile is quickly becoming large. Notice that the commands
for building the object files are the same except for the filenames. If we had to
change the command that was run for creating object files, we would have to
change each one of the rules. We can use pattern rules to make our makefile more
compact. Pattern rules provide a way to specify a common rule for building files
of one type from another. In our example we need to define a common rule for
generating object files (.o) from source files (.c) which will replace the four
separate rules in our previous listing. In our makefile, it will look like.
test_link: test_link.c libmymath.so
gcc -o test_link test_link.c libmymath.so

libmymath.so: libcubed.o libpowerfour.o libpowerfive.o libsquareroot.o

gcc -shared -Wl,-soname,libmymath.so -o libmymath.so \
libcubed.o libpowerfour.o libpowerfive.o libsquareroot.o

%.o: %.c
gcc -fPIC -c $<
The target %.o means any target with an extension .o, the portion before the
extension being called the stem. The corresponding prerequisite %.c means the
file with the same stem but with the extension .c. When a target that matches
this rule is being built, the automatic variable $< in the build command expands
to the prerequisite. That is if the rule is being applied to target: prerequisite
combination libcubed.o: libcubed.c, $< expands to libcubed.c (see auto-
matic variables below).

1.1.2 Variables
Notice that in the previous makefile, the names of the prerequisite object files for
the libmymath.so target is repeated in the command. We can reduce repetition
by using variables. Variables are created by simple assignment and referenced
by enclosing the variable name in $().

4
OBJECTS = libcubed.o libpowerfour.o libpowerfive.o libsquareroot.o

test_link: test_link.c libmymath.so

gcc -o test_link test_link.c libmymath.so

libmymath.so: $(OBJECTS)
gcc -shared -Wl,-soname,libmymath.so -o libmymath.so $(OBJECTS)

%.o: %.c
gcc -fPIC -c $<
This more convenient because whenever we add a function to libmymath.so, we
just need to add the object file name to the variable OBJECTS and it is included
in the build process. We can incorporate other automatic variables into our
makefile. Some of them are (for a complete listing see the GNU Make Manual).

• $@
Expands to the filename of the target
• $<
Expands to the first file in the prerequisites

• $^
Expands to the all files in the prerequisites
Our makefile will be
OBJECTS = libcubed.o libpowerfour.o libpowerfive.o libsquareroot.o

test_link: test_link.c libmymath.so

gcc -o $@ $^

libmymath.so: $(OBJECTS)
gcc -shared -Wl,-soname,$@ -o $@ $(OBJECTS)

%.o: %.c
gcc -fPIC -c $<

1.1.3 Phony Targets

A phony target is one whose target and prerequisites are not files that need
to be updated or created. A common phony target is clean, which removes
object files and executables that were created during a build so that the source
directory is ready for a complete rebuild of all files. The rule in our example
might look like.
clean:
rm *.o *.so test_link

Running the command

make clean

5
will cause make to build the target clean. Since no file named clean exists,
the commands for the rule are executed. As these commands do not create any
file named clean, this target will never be up to date and the commands will
always be executed if this target is specified for make. A problem that could
arise with such a rule is that, if a file named clean happend to exist, make would
assume that the target clean is up to date, since it has no prerequisites and
the commands would never be executed even if clean is specified as the target.
To avoid this, we can tell make that a target is not a real file. This is done by
adding the special target .PHONY and listing our phony target (here clean) as
a prerequisite. Thus we get.
.PHONY: clean

clean:
rm *.o *.so test_link
The final makefile for our example with the clean target included will be.
OBJECTS = libcubed.o libpowerfour.o libpowerfive.o libsquareroot.o

test_link: test_link.c libmymath.so

gcc -o $@ $^

libmymath.so: $(OBJECTS)
gcc -shared -Wl,-soname,$@ -o $@ $(OBJECTS)

%.o: %.c
gcc -fPIC -c $<

clean:
rm *.o *.so test_link

2 Installing Programs from Sources

Most application programs for Linux are distributed as package files which can
be easily installed and removed from the system by using utilities provided by
the particular distribution of Linux. For example, Ubuntu and Debian distribute
programs as Debian package files (.deb files) and RedHat distributes them in
the RedHat Package Manager format (.rpm files). Many times we might find
that an application or library is not available in one of these formats, either
because it is very new and hasn’t been packaged in one of these formats yet, or
the developer distributes the software only as source.

2.1 Tar Archives

Sources are usually distributed as a compressed tar archive. These are files
usually have the extension .tar.gz, .tgz or tar.bz2. When these archives
are extracted, you will obtain a directory with source code files among others,
including files with installation instructions, configuration scripts and makefiles.
In order to extract an archive, we must first check whether the archive will
extract into a separate directory or will place all files in the current directory.

6
We prefer the former because placing files in our current directory will mean
confusion with files that are already present there. In order to do this we use
the command.
tar -tzvf archive.tar.gz | less
or
tar -tjvf archive.tar.bz2 | less
The option -t tells tar not to extract the files but only to display the contents
of the archive. If you get output like.
-rw-r--r-- ershaad/ershaad 132383 2012-04-23 11:31 esc/1_os/os.pdf
-rw-r--r-- ershaad/ershaad 15789 2012-04-23 11:31 esc/1_os/os.odt
-rw-r--r-- ershaad/ershaad 17317 2012-04-23 11:31 esc/1_os/os.tex
-rw-r--r-- ershaad/ershaad 17320 2012-04-23 11:31 esc/1_os/os.tex.bak
-rw-r--r-- ershaad/ershaad 1706 2012-04-23 11:31 esc/1_os/os.aux
-rw-r--r-- ershaad/ershaad 157 2012-04-23 11:31 esc/todo.txt
drwxr-xr-x ershaad/ershaad 0 2012-04-23 11:31 esc/examples/
drwxr-xr-x ershaad/ershaad 0 2012-04-23 11:31 esc/examples/2_shell_scripting/

which shows that every file is rooted to a common directory (esc here),
then you are good to go and can extract the files, tar will create the directory
structure that matches the output we saw in the last output listing. If a directory
with the same name already exists, then the files are extracted there and will
overwrite files existing files with the same name.
If the output of tar -tjvf archive.tar.bz2 | less is like
-rw-r--r-- ershaad/ershaad 17320 2012-04-23 11:36 ./1_os/os.tex.bak
-rw-r--r-- ershaad/ershaad 1706 2012-04-23 11:36 ./1_os/os.aux
-rw-r--r-- ershaad/ershaad 157 2012-04-23 11:36 ./todo.txt
drwxr-xr-x ershaad/ershaad 0 2012-04-23 11:36 ./examples/
drwxr-xr-x ershaad/ershaad 0 2012-04-23 11:36 ./examples/2_shell_scripting/
-rwxr-xr-x ershaad/ershaad 30 2012-04-23 11:36 ./examples/2_shell_scripting/uniquefruits2.sh
-rw-r--r-- ershaad/ershaad 106 2012-04-23 11:36 ./3_awk_sed/examples/text_repeat.txt
-rwxr-xr-x ershaad/ershaad 149 2012-04-23 11:36 ./3_awk_sed/examples/vartot.awk
-rw-r--r-- ershaad/ershaad 33558 2012-04-23 11:36 ./3_awk_sed/awk_sed.tex.bak
drwxr-xr-x ershaad/ershaad 0 2012-04-23 11:36 ./2_shell_scripting/
-rw-r--r-- ershaad/ershaad 99 2012-04-23 11:36 ./2_shell_scripting/Makefile

where all the files are not rooted to a common directory, then we need
to create a suitable named directory using mkdir, move the archive into that
directory using mv and then finally extract it.
Archives are extracted by using the commands
tar -zxvf archive.tar.gz
or
tar -zyvf archive.tar.bz2

2.2 Documentation
Once you’ve extracted the files in the archive (commonly referred to as a tarball ),
you will most probably find files named similar to README or INSTALL. It is a
very good idea to read these first before proceeding with the installation since
they usually contain important information and instructions for compiling and
installing the package.

7
2.3 The configure script
Most sources will contain a script called configure. Its job is to obtain in-
formation about you system and software and the create a makefile that will
contain instructions to build the software for your system.
The next step therefore after extracting the tarball is to run the configure
script. Usually it is simply run with the command.
./configure

Depending on the actual package the configure script can take options that
modify how the program built. For example, there can be options to specify
which feature are included in the final executables, or which specific versions of
libraries the program will be linked to. All options provided by the configure
script can be see by running.

./configure --help

2.3.1 The --prefix Option

The --prefix option to configure is used to set where the files that are built
are place on the system. The default is to place them in locations such that
the program can be used by any user on the system. The --prefix option can
be used to specify an alternate directory under which the files will be installed.
For example
./configure --prefix=/home/ershaad/local

2.4 Compilation
After configure completes successfully, makefiles are generated. The actual
compilation of the package can now begin by issuing the make command. Some-
times the documentation might specify alternate targets to either build certain
portions or the package or to perform some function, like testing the built files
for correctness for instance. For example.

make all
or
make check

2.5 Installation
Once the make command successfully runs, you should have the final program
(executables, libraries, man pages, etc.) ready for copying to their correct loca-
tions on the system. This is done by running

make install
If you have not specified a --prefix, the default location for installation are
system-wide locations. In order to write to these locations you will need to
be the root user. Thus before executing make install, you can change to
the root (if you have the password and are allowed to) user by typing the su

8
command. On some systems the command sudo make install can be run.
This automatically executes make as the root user, if you are allowed by the
administrator to use the sudo command.
Once make install has successfully run, the program installation is com-
plete. The installation of programs can be more complicated than the usual
case described above. This is especially true when the package that is being
built from sources depends on libraries. Make sure to read the documentation.