Processes and Services

Chris Kuehl
Who am I? ckuehl@ocf.berkeley.edu

Infrastructure Engineer @ Yelp

Formerly Berkeley / OCF

What is this lecture about?
Chapter 1: Processes
Chapter 2: Init systems & services
CHAPTER ONE
Processes
What is a process?
It’s just ~one running
program.
Each process has its
own memory space,
threads, code, etc.
Process hierarchy
Each process has one
parent, typically the
process that started it.
Processes can have
many children.
Processes vs threads
Both methods of running code “at the same time”
A process is a collection of one or more threads
Threads are all part of the same “program”, share
the same memory
Processes communicate over some interface
So why is my laptop running
6000 chrome processes?
How are processes created?
A process “forks” into two separate processes
using the fork syscall
The parent keeps its PID, the child gets a new
PID.
How many “hello”s get printed?
int main(void) {
fork();
printf("hello: %d\n", getpid());
}
How many “hello”s get printed?
Process ID 1000
int main(void) {
fork();
printf("hello: %d\n", getpid());
}

Process ID 1000 Process ID 2000

int main(void) { int main(void) {
fork(); fork();
printf("hello: %d\n", getpid()); printf("hello: %d\n", getpid());
} }
How many “hello”s get printed?
Process ID 1000
int main(void) { $ ./foo
fork();
printf("hello: %d\n", getpid()); hello: 27951
}
hello: 27952

Process ID 1000 Process ID 2000

int main(void) { int main(void) {
fork(); fork();
printf("hello: %d\n", getpid()); printf("hello: %d\n", getpid());
} }
How many “hello”s get printed?
int main(void) {
fork();
fork();
fork();
printf("hello: %d\n", getpid());
}
How many “hello”s get printed?
$ ./foo
int main(void) { hello: 27956
fork(); hello: 27958
fork(); hello: 27959
fork(); hello: 27957
printf("hello: %d\n", hello: 27961
getpid());
} hello: 27960
hello: 27963
hello: 27962
How to make the child do something
else? Process tree after start:

pid_t result = fork();

if (result > 0) {
/* parent process */
sleep(100);
} else if (result == 0) {
/* child process */
sleep(10);
} else { /* error */ } child exits first
What happens when processes
exit? Process tree after start:

pid_t result = fork();

if (result > 0) {
/* parent process */
sleep(100);
} else if (result == 0) { After 10 seconds:
/* child process */
sleep(10);
} else { /* error */ }

????
Zombie processes
When a process exits, it
temporarily becomes a zombie:

● No longer running

● Still exists in process table so

parent can collect exit status,
uses no other resources

● Totally normal!
What happens when a
process dies? Processes always exit with
an “exit code” (int between
pid_t result = fork(); 0 and 255)
if (result > 0) {
/* parent process */ Typically, 0 is success,
sleep(100); anything else is an error
} else if (result == 0) {
/* child process */
exit(123);
} else { /* error */ }
Parents need to wait() on children
(aka reap)
pid_t result = fork(); Process tree after start:
if (result > 0) {
/* parent process */
int status;
wait(&status);
printf("%d\n", WEXITSTATUS(status));
} else if (result == 0) { After 10 seconds:
/* child process */
sleep(10);
exit(123); Prints out:
} else { /* error */ }
What if the parent exits first?
pid_t result = fork();
Process tree after start:
if (result > 0) {
/* parent process */
parent
sleep(1);
child
exit(0);
} else if (result == 0) { After 1 second:
/* child process */
sleep(10);
child
exit(123);
} else { /* error */ }
Orphaned processes are
re-parented under PID 1
Process signaling
Simple messages to other processes.

● SIGTERM: “please exit”

● SIGINT: interrupt, generated by ^C in the terminal
● SIGKILL: kill immediately without a chance to clean up
● SIGWINCH: terminal was resized
● SIGHUP: terminal was closed
● SIGSTOP / SIGCONT: pause/resume

Typically send using kill or pkill, e.g. kill -9 <pid> or

pkill -HUP apache2.
How do processes communicate?
All kinds of inter-process communication (IPC):

● Exiting with a status code

● Signals (e.g. SIGTERM, SIGKILL, SIGHUP)
● Pipes (just like stdin, stdout, stderr)
● Network (e.g. TCP or UNIX sockets)
● Message bus (e.g. dbus)
● ...and tons more...
Working with processes: ps
Nobody remembers what the arguments to ps mean, just
memorize a few you find useful :)
ps aux ps xawuf ps -o pid,uname,comm
Working with processes: pstree
Working with processes: pgrep & pkill

(pgrep is
especially useful
for composing)
Working with processes: htop
(makes pretty screenshots)
CHAPTER TWO
Init systems & services
What is an init system?
The first process launched (PID 1), lives forever
Core responsibilities:
● Launching enough processes to make the
system useful
● Reap orphaned zombie processes
What does a bare-bones init
system look like?
/bin/bash can work in a pinch

What about in code?

Bare-bones init
pid_t result = fork(); reaps all children,
if (result > 0) { even those it didn’t
/* parent process */ spawn (orphans)
while (wait(NULL) != result) {}
} else if (result == 0) {
/* child process */
execl("/bin/bash", "bash"); // any process here
} else { /* error */ }
spawns some useful
process
Traditional init systems
Even traditional init systems do more than the minimum
required.

Typical extra responsibilities:

● coordinating startup/shutdown (often via “runlevels”)

● launching services
● handling signals
Different kinds of processes
Not strict definitions:
● Interactive: foreground processes like
chrome, vim, htop

● Daemons: background processes like sshd,

httpd, rsyslogd, nginx, postfix
How do we control services?
Lots of ways…
● We could signal processes (e.g. kill -HUP)
● We could use their CLIs (rndc, apachectl,
etc.)
● We could rely on our init systems
Let’s look at how init systems do it...
“Old”-style init scripts
Turing-complete, actual scripts:
/etc/init.d/$service $action

Each script is responsible for keeping track of processes.

● Typically PID files (can be stale, race-prone)

● Inconsistent actions provided (start/stop/restart,
reload, graceful, status, etc.)
● Inconsistent behavior and output (e.g. status command)
● Incredibly annoying to write, often copy-pasted
“Old”-style init scripts
/etc/init.d/nginx on ubuntu
What would modern service supervision look like?
Traditional Modern
Scripts that can do anything Declarative config files
Store state using PID files Store state in the init system
Inconsistent actions (each Consistent actions via
init script its own CLI) common CLI
Difficult to control ordering Ordering via dependencies
or events
Let’s look at some examples...
Modern init system: upstart
Developed by Canonical (Ubuntu), used for a
few releases
Event-based (services subscribe to events)
Parallel start and stop
Way easier to write than
traditional init scripts
Modern init system: upstart
/etc/init/my-cool-server.conf

description "my cool server"

start on filesystem and started networking

stop on shutdown

exec /usr/bin/my-cool-server
Modern init system: systemd
Originally comes from Fedora-land
Dependency-based
Parallel start and stop
Way easier to write than traditional init scripts
Modern init system: systemd
Goals:
● Unify core Linux tooling across distributions
● Provide excellent UX
Ended up reimplementing syslog, cron,
networking, GRUB, ConsoleKit, udev
Turns out they’re controversial
In practice, systemd has won
...
systemctl status

systemctl start/stop
systemctl enable/disable
systemctl (no arguments)
journalctl
View recent logs from a unit:
journalctl -u $unit -e

Tail logs from a unit:

journalctl -u $unit -f
Writing systemd unit files
/etc/systemd/system/my-cool-server.service
[Unit]
Description=my cool server
Requires=network-online.target opt.mount
After=network-online.target opt.mount

[Service]
ExecStart=/usr/bin/my-cool-service

[Install]
WantedBy=multi-user.target
Conclusion (i.e. opinions)
● Think about the UNIX philosophy as a guiding
principle, not a law

● Focus on the health of entire systems

● How can you be most productive & design

things reliably?
Signin

decal.ocf.berkeley.edu/signin
Week 6: Engelbart