Linux Troubleshooting Tools

’19H2

송인식
 strace
• strace is a system call trace
– Debugging tool which prints out a trace of all system calls
made by a process/program
– Program to be traced need not be recompiled for this, so it
can be used on binaries for which there is no source
• In the simplest case, strace runs the specified
command until it exits
• Intercepts and records the system calls which are
called by a process and the signals which are received
by a process

Linux Troubleshooting Tools 2

 strace
• The name of each system call, its arguments and its
return value are printed to standard error or to the file
specified with the -o option
• Each line in the trace contains the system call name,
followed by its arguments in parentheses and its
return value

Linux Troubleshooting Tools 3

 strace
• To get information about:
– Which config files really were read?
– Which was the last file or shared library read before your
program crashed?
– How often is a system call executed?
– How much time does a kernel call consume?
– How much compute time does the program itself use
between kernel calls?

Linux Troubleshooting Tools 4

 strace
• Examples

Linux Troubleshooting Tools 5

 /proc File System
• A special filesystem in Unix-like operating systems that
presents information about processes and other
system information in a hierarchical file-like structure
• Acts as an interface to internal data structures in the
kernel.
• Used to obtain information about the system and to
change certain kernel parameters at runtime (sysctl).

Linux Troubleshooting Tools 6

 /proc/PID
Directory Description
/proc/PID/cmdline Command line arguments.
/proc/PID/cpu Current and last cpu in which it was executed.
/proc/PID/cwd Link to the current working directory.
/proc/PID/environ Values of environment variables.
/proc/PID/exe Link to the executable of this process.

/proc/PID/fd Directory, which contains all file descriptors.

/proc/PID/maps Memory maps to executables and library files.

/proc/PID/mem Memory held by this process.
/proc/PID/root Link to the root directory of this process.
/proc/PID/stat Process status.
/proc/PID/statm Process memory status information.
/proc/PID/status Process status in human readable form.
Linux Troubleshooting Tools 7
 Other Files in /proc
File Description
/proc/crypto list of available cryptographic modules
nformation (including device numbers) for each of the l
/proc/diskstats
ogical disk devices
list of the file systems supported by the kernel at the ti
/proc/filesystems
me of listing
/proc/kmsg holding messages output by the kernel
/proc/meminfo summary of how the kernel is managing its memory.
information about any devices connected via a SCSI or
/proc/scsi
RAID controller
/proc/tty information about the current terminals
containing the Linux kernel version, distribution number
, gcc version number (used to build the kernel) and any
/proc/version
other pertinent information relating to the version of th
e kernel currently running
Linux Troubleshooting Tools 8
 gdb -GNU Debugger
• Purpose
– allows you to see what is going on inside another program
while it executes
• You can do!
– start program, specifying anything that might affect its
behavior
– stop program with special condition
– when program stopped, watch out function
– change things in your program

Linux Troubleshooting Tools 9

 Using gdb
• To start gdb with your hello program type:
gdb HelloProg
• When gdb starts, your program is not actually running.
• You have to use the run command to start execution.
• Before you do that, you should place some break
points.
• Once you hit a break point, you can examine any
variable.

Linux Troubleshooting Tools 10

Useful gdb commands

Linux Troubleshooting Tools 11

 ELF: Executable and Linking Format
• The a.out format served the Unix community well for o
ver 10 years.
• However, to better support cross-compilation, dynamic
linking, initializer/finalizer (e.g., the constructor and des
tructor in C++) and other advanced system features, a.
out has been replaced by the elf file format.
• Elf stands for “Executable and Linking Format.”
• Elf has been adopted by FreeBSD and Linux as the curr
ent standard.

Linux Troubleshooting Tools 12

 Source Translation
User-created files

C/C++
C/C++Source
Source Assembly
Assembly Linker
Makefile and Header
and Header Source
Source Script
Files
Files Files
Files File

Make Utility preprocessor

compiler assembler

Object
Object
Archive Utility Files
Files

Library
Library
Files
Files
Linker and Locator

Shared
Linkable Executable Link Map
Object
Image File Image File File
File

Linux Troubleshooting Tools 13

 Linking

Linux Troubleshooting Tools 14

 Useful Tools
• strace: The strace tool traces the system calls, special
functions that interact with the operating system. You
can use this for many types of problems, especially
those that relate to the operating system.
• ltrace: The ltrace tool traces the functions that a
process calls. This is similar to strace, but the called
functions provide more detail.
• lsof: The lsof tool lists all of the open files on the
operating system (OS). When a file is open, the OS
returns a numeric file descriptor to the process to use.
This tool lists all of the open files on the OS with their
respective process IDs and file descriptors.

Linux Troubleshooting Tools 15

 Useful Tools
• top: This tool lists the “top” processes that are running
on the system. By default it sorts by the amount of
current CPU being consumed by a process.
• traceroute/tcptraceroute: These tools can be used to
trace a network route (or at least one direction of it).
• ping: Ping simply checks whether a remote system can
respond. Sometimes firewalls block the network
packets ping uses, but it is still very useful.
• hexdump or equivalent: This is simply a tool that can
display the raw contents of a file.

Linux Troubleshooting Tools 16

 Useful Tools
• tcpdump and/or ethereal: Used for network problems,
these tools can display the packets of network traffic.
• GDB: This is a powerful debugger that can be used to
investigate some of the more difficult problems.
• readelf: This tool can read and display information
about various sections of an Executable and Linking
Format (ELF) file.

Linux Troubleshooting Tools 17

Questions?

Linux Troubleshooting Tools 18