P. 1
Solaris 10 Administration Topics Workshop 1- Administration

Solaris 10 Administration Topics Workshop 1- Administration

1.0

|Views: 1,141|Likes:
Published by dev5878
Solaris Administration Topics - Administration
Solaris Administration Topics - Administration

More info:

Published by: dev5878 on Feb 23, 2010
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

12/29/2012

pdf

text

original

Sections

#!/usr/sbin/dtrace -s
#pragma D option flowindent
pid$1::$2:entry
{
self->trace = 1;
}
pid$1:::entry, pid$1:::return, fbt:::
/self->trace/
{
printf("%s", curlwpsinfo->pr_syscall ?
"K" : "U");
}
pid$1::$2:return
/self->trace/
{
self->trace = 0;
}

147

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

148

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

DTrace Toolkit

DTrace Toolkit with lots (> 90) of great scripts
Includes scripts for Python, Perl, Java, PHP, Ruby, Tcl,
Javascript
Best starting point for learning DTrace

http://www.opensolaris.org/os/community/
dtrace/dtracetoolkit/

149

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

DTrace Toolkit Hits

dexplorer - run a lot of tools for a few
seconds and log output to a file
Other key scripts include

dtruss, dvmstat, execsnoop,
hotkernel, hotuser, errinfo,
iopattern, iosnoop, iotop,
opensnoop, procsystime,
rwsnoop, rwtop, statsnoop

150

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

DTrace Lab Preliminary Steps

Get a Solaris 10 machine
Become the root user
Make a new directory – use it to save all the
examples from this talk
Might want to record your command-line
history for future reference
In a zone, DTrace has limited functionality so
not all examples will work

151

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Introduction to DTrace

Quick start overview of DTrace and
programming basics
Listing probes
Enabling probes
Built-in variables
The
trace(), and printf() actions
Then more actions, examples, fun!

152

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

What is a Probe?

Provided by a “provider”
Many providers, including
syscall - system calls entry / return
vminfo - virtual memory stats
sysinfo - sysinfo stats
io - disk and NFS events
sched - system scheduling events
profile - fixed sampling
dtrace - BEGIN / END probes
pid - user-level tracing
fbt - raw kernel function entry / return tracing

153

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Probe Names

provider:module:function:name
Some providers don’t use all 4 fields
Leave any field blank to match all
Fundamentally - determine proper provider then
check the DTrace Guide
If using “syscall” provider, check man(2)
If using “pid” provider check program source code
and man(3C)
If using “fbt” provider check
http://
cvs.opensolaris.org/source/

154

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Listing Probes

Use dtrace -l to list all probes
Use
dtrace -lv to list all probes with attributes
Use
dtrace -lP to list probes for specific provider
Can mix
-l with -n to list probes matching a pattern
Specify probes by a four-tuple:

provider:module:function:name
Any component can be blank
Exercise: list some probes
Exercise: combine
-l and -n
Exercise: try using wildcards and grep for the various components of
the probe tuple

155

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Enabling Probes

Try it:

#dtrace -n syscall:::entry

Note here that “dtrace” is the “consumer”,
and a probe is specified
Traces every system call on the system
What does the output mean?
Exercise: trace a
single system call entry

156

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

The trace() Action and Variables

Use the trace() action to trace any datum

e.g. results of computation, variables, etc.

Try tracing a value:

#dtrace -n 'syscall:::entry { trace(10); }'
#dtrace -n 'syscall::fork*: { trace(pid); }'

Exercise: trace a variable

execname – currently running executable

timestamp – nanosecond timestamp

walltimestamp – seconds since the Unix epoch

pid, uid, gid, etc. – what you'd expect

157

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Tracing Process Creation

#dtrace -n 'syscall::exec*:return { trace
(execname) ; }’

Try at “entry” rather than return - what happens?

158

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

D Program Structure

probe-
description

{
action;
action;
...
}

159

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Predicates

Predicates are arbitrary D expressions that
determine if a clause is executed
Specify a predicate like this:

/arbitrary-expression/

Try limiting tracing to a particular executable

#dtrace -n 'syscall:::entry/execname ==
“Xorg”/{}'

Exercise: mix predicates and the trace() action

160

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

D Program Structure

probe-
description
/predicate/

{
action;
action;
...
}

161

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

More Variables

Each part of a probe has an associated variable

probeprov – provider name

probemod – module containing the probe (if any)

probefunc – function containing the probe

probename – name of probe

Probes can have arguments (arg0, arg1, etc.)

Different for each provider and each probe

syscall entry probe arguments are the parameters passed to the
system call
Exercise: try tracing system call arguments

162

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

The printf() Action

Modeled after printf(3C) – behaves as you'd expect
Small difference: 'l's (“ell”) not needed to specify
argument width – but you
can use them
Exercise: use
printf to trace the pid and execname
Done? Try out your favorite printf() format
characters

163

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Fun With walltimestamp

The printf() action has some additional
format characters (some borrowed from
mdb(1))
%Y can be used to format a date
Try it:

#dtrace -n 'BEGIN{ printf(“%Y”,
walltimestamp); }'

164

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

D-Scripts

Can do everything from the command-line
Big DTrace enabling can become confusing
Put them in an executable script:

#!/usr/sbin/dtrace -s

syscall:::entry
{

trace(execname);

}

Exercise: try it – make it executable
Quiet mode
dtrace -q
Use \n to terminate each line when quiet mode

165

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Aggregations

Often the individual data points are overwhelming
Aggregations provide a way of accumulating data

Done at the data source, so very efficient

Data stored efficiently on MP systems
Several aggregating functions
Aggregations can be keyed by an arbitrary tuple of D expressions

Think of them as associative arrays

By default, the contents of aggregations are printed when the consumer
completes

e.g. when you hit ^C

166

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Simple Aggregation With

count()

Aggregations are specified like this:

@name[arbitrary-tuple] = action(arguments)

The name and tuple may be omitted
The arguments depend on the aggregating action
Try it:

# dtrace -n 'syscall:::entry{ @ = count(); }'

Exercise: try specifying a name for the aggregation
Exercise: try adding tuple keys (comma separated)
Exercise: produce a count for each system call

167

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

The quantize() Aggregating
Action

The quantize() action is particularly useful for
performance work
Takes a single numeric argument
Produces a histogram in power of two buckets
Try it:

# dtrace -n 'syscall::write:entry{ @ =
quantize(arg2); }'

168

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Using DTrace as Non-Root

DTrace use controlled by three least-privilege
bits

dtrace_proc – process access
dtrace_user – user space access
dtrace_kernel – kernel access

Note that giving the dtrace_kernel proc to a
user is (almost) equivalent to giving them root

169

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

DTrace and Zones

As of Nevada 37, DTrace can run inside non-global
zones

Note that zones can have proc and user but not kernel

# zonecfg -z myzone
zonecfg:myzone> set
limitpriv=“default,dtrace_proc,dtrace_use
r”
zonecfg:myzone> exit
# zoneadm -z myzone boot
# zlogin myzone
myzone# dtrace -l
...

170

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

DTrace One-Liners

Snarfed from http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_One_Liners

Processes

* New processes with arguments,

dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

Files

* Files opened by process name,

dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

* Files created using creat() by process name,

dtrace -n 'syscall::creat*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

Syscalls

* Syscall count by process name,

dtrace -n 'syscall:::entry { @num[execname] = count(); }'

* Syscall count by syscall,

dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'

* Syscall count by process ID,

dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

* Read bytes by process name,

dtrace -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }'

I/O

* Write bytes by process name,

dtrace -n 'sysinfo:::writech { @bytes[execname] = sum(arg0); }'

* Read size distribution by process name,

dtrace -n 'sysinfo:::readch { @dist[execname] = quantize(arg0); }'

* Write size distribution by process name,

dtrace -n 'sysinfo:::writech { @dist[execname] = quantize(arg0); }'

Physical I/O

* Disk size by process ID,

dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'

* Disk size aggregation

dtrace -n 'io:::start { @size[execname] = quantize(args[0]->b_bcount); }'

* Pages paged in by process name,

dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'

171

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

More DTrace One-liners

Memory

* Minor faults by process name,

dtrace -n 'vminfo:::as_fault { @mem[execname] = sum(arg0); }'

User-land

* Sample user stack trace of specified process ID at 1001 Hertz

dtrace -n 'profile-1001 /pid == $target/ { @num[ustack()] = count(); }' -p PID

* Trace why threads are context switching off the CPU, from the user-land perspective,

dtrace -n 'sched:::off-cpu { @[execname, ustack()] = count(); }'

* User stack size for processes

dtrace -n 'sched:::on-cpu { @[execname] = max(curthread->t_procp->p_stksize);}'

Kernel

* Sample kernel stack trace at 1001 Hertz

dtrace -n 'profile-1001 /!pid/ { @num[stack()] = count(); }'

* Interrupts by CPU,

dtrace -n 'sdt:::interrupt-start { @num[cpu] = count(); }'

* CPU cross calls by process name,

dtrace -n 'sysinfo:::xcalls { @num[execname] = count(); }'

* Trace why threads are context switching off the CPU, from the kernel perspective,

dtrace -n 'sched:::off-cpu { @[execname, stack()] = count(); }'

* Kernel function calls by module

dtrace -n 'fbt:::entry { @calls[probemod] = count(); }'

172

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

A Couple of My D Scripts

#!/usr/sbin/dtrace -s
#pragma D option flowindent
pagefault:entry
{

self->mytime=timestamp;
self->t = 1;

}
pagefault:return
/self->t == 1/
{

@[probename] = quantize (timestamp - self->mytime);
self->mytime=0;
self->t = 0;

}

173

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

A Couple of My D Scripts

#!/usr/sbin/dtrace -s
/* pass in a process ID, output is 1 line per file that
failed to open */
#pragma D option quiet
syscall::open*:entry
/ pid == $1 /
{

self -> filename = copyinstr(arg0);

}

syscall::open*:return
/errno != 0 && pid == $1 && strlen(self -> filename) /
{

printf("%s %s %d %s\n",execname,probefunc,errno, self-

>filename);
}

174

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

A Couple of My D Scripts

#!/usr/sbin/dtrace -s -q
/* for a given pid sum up the amount of time spent in each
system call */
syscall:::entry
/pid == $1/
{
self->tm = timestamp
}

syscall:::return
/self->tm/
{
@[probefunc] = quantize(timestamp - self->tm);
self->tm = 0
}

175

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Great DTrace Talk

By Benoît Chaffanjon, Sun Solutions Center
Good methodology, step-by-step examples, one-liners
Show all processes as created:

dtrace -n 'proc:::exec{printf("%s execing %s, ,
uid/zone =%d/%s\n",execname,args[0],uid,zonename)}'

Watch for errors (they slow thread execution):

/usr/sbin/dtrace -qn 'syscall:::return /errno != 0 && pid != $pid/
{ \@Errs[execname,probefunc,errno] = count(); }
dtrace:::END {printa("%s %s %d %\@d\\n",\@Errs); }'

Watch for errors (they slow thread execution):

/usr/sbin/dtrace -qn 'syscall:::return /errno != 0 && pid != $pid/
{ \@Errs[execname,probefunc,errno] = count(); }
dtrace:::END {printa("%s %s %d %\@d\\n",\@Errs); }'

176

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Great DTrace Talk (cont)

Check number of threads in a process

profile:::profile-100hz /pid/{@[pid, execname] = lquantize(cpu, 0, 512, 1);}

Who is causing xcalls

dtrace -q -n 'xcalls { @[pid,tid,execname] = count(); }'

Full talk at: http://opensolaris.org/os/project/sdosug/past_meetings/
Performance_Analysis_Using_DTrace.pdf

177

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Chime

First step toward a GUI for DTrace
Available open source at
http://opensolaris.org/os/
project/dtrace-chime/

178

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Sun Studio Compilers

Some very nice tools, including DProfile
and D-Light
D-Light integrated D-Trace tool in Sun
Studio 12 and above
GUI and Graphic

http://developers.sun.com/sunstudio/downloads/express/index.jsp

179

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

D-Light

180

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Ortera (commercial)

“A visual DTrace for storage”
Runs on SPARC S8, 9, 10; S10x86 servers
Sol, Win, Linux, Mac client
Haven’t tried it myself but have heard good
things

http://www.ortera.com/products/index.html

181

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

DTrace Tutorial

“Context Switch” - UK consulting and
training company.
Have a DTrace workshop and put course
on-line

http://www.context-switch.com/
performance/dtrace.htm

182

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Other Old and New Important Performance Tools

183

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

prstat

“Best” process state tool
Updates like top
-m microstate accounting
-L light-weight processes info
-Z zone-based info
-t per-user info
-T task info
-v verbose process usage

184

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

fsstat

New as of Nevada 36
Per file system statistics!

# fsstat -F

new name name attr attr lookup rddir read read write write
file remov chng get set ops ops ops bytes ops bytes
12.3K 1.48K 949 683K 8.94K 5.32M 16.8K 555K 266M 2.90M 2.58G ufs
0 0 0 2.51K 0 4.14K 1.66K 1.60K 313K 0 0 proc
0 0 0 83 1 117 2 8 19.5K 0 0 nfs
76.6K 15 3.36K 1.81M 268K 3.54M 1.58K 6.33K 15.4M 207K 1.26G zfs
0 0 0 21.3K 0 116K 843 323K 455M 0 0 hsfs
0 0 0 32.4K 0 0 0 0 0 0 0 lofs
4.07K 2.87K 742 23.3K 366 8.28K 78 51.9K 51.3M 42.7K 48.5M tmpfs
0 0 0 829 0 0 0 23 2.73K 0 0 mntfs
0 0 0 0 0 0 0 0 0 0 0 nfs3
0 0 0 0 0 0 0 0 0 0 0 nfs4
0 0 0 4 0 0 0 0 0 0 0 autofs

185

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

trapstat

# trapstat
vct name | cpu0
------------------------+---------
24 cleanwin | 158
41 level-1 | 101
44 level-4 | 9
46 level-6 | 2
49 level-9 | 100
4a level-10 | 100
4e level-14 | 101
60 int-vec | 11
64 itlb-miss | 353
68 dtlb-miss | 1701
6c dtlb-prot | 3
84 spill-user-32 | 89
8c spill-user-32-cln | 14
98 spill-kern-64 | 854
a4 spill-asuser-32 | 34
ac spill-asuser-32-cln | 94
c4 fill-user-32 | 94
cc fill-user-32-cln | 101

186

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

pmap -x

# pmap -x 656
656: -sh

Address Kbytes RSS Anon Locked Mode Mapped File
00010000 288 200 - - r-x-- dev:136,8 ino:

2613

00066000 16 16 - - rwx-- dev:136,8 ino:

2613

0006A000 16 16 - - rwx-- [ heap ]
FFBFE000 8 8 - - rw--- [ stack ]
-------- ------- ------- ------- -------
total Kb 328 240 - -

187

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

mdb ::memstat

# mdb -k
Loading modules: [ unix krtld genunix ip usba s1394
ufs_log nfs random ptm login dmux cpc lofs ]

> ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 3604 28 6%
Anon 5941 46 10%
Exec and libs 2442 19 4%
Page cache 2028 15 3%
Free (cachelist) 2815 21 5%
Free (freelist) 45272 353 73%

Total 62102 485
>

188

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

kstat

Displays lots and lots of kernel statistics
Mostly useful when working with Sun to
debug a problem, as most of the displayed
variables are undocumented

189

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

lockstat

Obscure, but essential in some
circumstances

Especially for kernel engineers

Shows kernel locks, timings
Useful to determine system slowness causes
when all else fails
Improved with even more options in Solaris 8
Rewritten in S10 as part of DTrace

190

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

lockstat (cont)

Can get kernel profiling as well

# lockstat –I 997 –i sleep 120

191

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Solaris Resource Manager

<=S8 commercial Sun program
Goes beyond partitioning
More fine-grain control
Virtual memory control as well
Good for server consolidation
Full, better version now included with S9

192

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Solaris Resource Manager - 1

Really a bunch of disparate facilities
Resource limitations
Based on projects, tasks, processes
Controlled by /etc/project
Limited resources that can be limited

IP QOS
Fair share scheduler

193

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Solaris Resource Manager - 2

Fair Share Scheduler is cool
Shares given out for a domain (projects,
tasks, processes)
Each object gets (its share) / (total
share) worth of the system if it wants it
Else resources used as available /
requested

194

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Solaris Resource Manager - 3

Enable FSS on the system
#
dispadmin -d FSS
Check the default class with dispadmin -d
Move all existing processes to FSS scheduling class
#
priocntl -s -c FSS -i class TS
Check scheduling classes in use:
#
ps -ef -o pset,class | grep -v CLS | sort |
uniq
- IA
- TS
- FSS
- SYS

Enter share information for each project in /etc/projects:

testproject:100::::project.cpu-shares=(privileged,10,none)

195

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Solaris Resource Manager - 4

Or set them dynamically:
#
prctl -r -n project.cpu-
shares -v 3 -i project
testproject

Launch a task in the project:

$ newtask -p testproject /usr/
tmp/cpuhog &
Use prstat to watch the system,
including project information

196

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Solaris Resource Manager (>= S 9)

Physical memory use now manageable via
rcapd daemon
Asynchronous resource limiter
Runs around kicking out pages to
reduce the working set, as dictated by
configuration
See “System Administration Guide: N1…”
Ch 10

197

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Resource Manager and /etc/system

Many items moved from /etc/system
Check app deployment guides to be sure
Note this is per zone the app is installed in
For example, Websphere guidance (
http://www.sun.com/software/
whitepapers/solaris10/websphere6_sol10.pdf
) shows to leave /etc/
system
alone and instead modify /etc/project to containing the following additions to
the resource controls for user.root. These project resource settings are read at login.

bash-3.00# cat /etc/project
system:0::::
user.root:1::::
process.max-file-descriptor=(privileged,1024,deny);
process.max-sem-ops=(privileged,512,deny);
process.max-sem-nsems=(privileged,512,deny);
project.max-sem-ids=(privileged,1024,deny);
project.max-shm-ids=(privileged,1024,deny);
project.max-shm-memory=(privileged,4294967296,deny)
noproject:2::::
default:3::::
group.staff:10::::

198

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Resource management and /etc/system (cont)

Now that we are using resource management we can
use some new features. For example notify via syslog if
these resources are used near their limit:
Also to enable warnings via syslog if the resource limits
are approached by executing the following commands
(they update the
/etc/rctladm.conf file):

# rctladm -e syslog process.max-file-descriptor
# rctladm -e syslog process.max-sem-ops
# rctladm -e syslog process.max-sem-nsems
# rctladm -e syslog process.max-sem-ids
# rctladm -e syslog process.max-shm-ids
# rctladm -e syslog process.max-shm-memory

199

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Things I Don’t Cover (yet)

dladm
shareadm
smbios
wificonfig

200

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Cool Commands (<= Nevada)

Profile of executable

truss –c

Show a process’ system calls and time they took

truss –D

Stop a process at a given system call

truss –T

Download pre-build open source packages

pkg-get (http://www.bolthole.com/solaris)

Manage core file creation and retention

coreadm

Reap zombie processes

preap

What is system memory being used by?

vmstat –p

What is a process using memory for?

pmap –x

201

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

T1 (Niagara) Performance

cooltst analyzes expected app performance
if app moved to T1 chip

http://cooltools.sunsource.net/cooltst/

cooltools suggests application performance
improvements on T1

http://cooltools.sunsource.net/cooltuner/index.html

202

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

What’s Next for Solaris?

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

Multiple Projects in the Works

Clearview - unify features implemented by network interfaces
Crossbow - network virtualization and resource control
NFS - pNFS, NFS server in non-global zone, DTrace for NFS v4
Network storage (FCOE, Infiniband)
OpenSolaris ports - easy install 3rd party software
Mem / VM resource management
ZFS encryption
Native CIFS client, server
Caiman - Solaris install experience
See them all at
http://www.opensolaris.org/os/projects

204

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

In the Liberal Arts Tradition

Philosophy

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

System Administration Best Practices
From March 2003 SysAdmin Magazine column
Consensus administration best practices
(Solaris and general) with contributions from
many experienced sysadmins
Contribute at

bestpractice@galvin.info

206

Topics

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

SysAdmin Best Practices (1)

Keep an Eye peeled and the wall at your back

Know how your systems run when no problems, put debugging
tools in place

Communicate with users

They can “help” spot problem, give you room to work when
trouble strikes

Help users fix it themselves

Knowledge transfer to fellows, users

Use Available Information

RTFM is right, after all these years, use available tech support

207

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

SysAdmin Best Practices (2)

Know when to use strategy, when to use tactics

Hand-to-hand combat vs. arranging the battlefield to increase
your odds of winning

All projects take 2X scheduled time and money

So 2 X estimates to prepare!

It’s not done until its tested

Great aggravation from untested changes

It’s not done until its documented

Decrease wheel-reinvention, miscommunication

Never change anything on Fridays…or Mondays

Speed kills, causes unhappy weekends

208

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

SysAdmin Best Practices (3)

Audit before Edit

Review system logs, understand state before making changes

Use defaults whenever possible

Too clever causes too complex

Always be able to undo what you are about to do

Copy individual files, directories, backup systems to disk/tape

Do not spoil management

Don’t let management put you in lose/lose situations

If you haven’t seen it work, it probably doesn’t

Discount the marketing, watch the details

If you’re fighting fires, find the source

Implement alarming, log file monitoring, push important data, don’t pull
unimportant

209

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

SysAdmin Best Practices (4)

If you don’t understand it, don’t play with it on production systems

Get a QA environment for experiments, before mistakes cost you in
production

If it can be accidentally used, and can produce bad consequences,
protect it

Put scripts around powerful commands or procedures, boxes around
power-off buttons

Occam’s Razor is very sharp

Check the simple stuff first, avoid complex solutions to simple problems

The last change is the most suspicious

Even if whatever changed couldn’t possibly be causing the current problem, it
probably is

When in doubt, reboot

Rebooting still solves problems, when used appropriately

210

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

SysAdmin Best Practices (5)

If it ain’t broke, don’t fix it

Consider how much time has been wasted by those who said “just one more
tweek”…

Save early and often

Don’t be the guy who lost his thesis when his floppy disk went bad

Dedicate a system disk (or 4)
Have a plan

Develop written task list, reuse it when task reoccurs or use as basis for
similar tasks

Cables and connectors can go bad

Be sure to check them, especially after board changes & system moves

Mind the power

Check power supplied vs. power drawn, grounding, single power grid vs.
multiple into a system
Same with cooling

211

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

SysAdmin Best Practices (6)

Try before you buy

If possible, the best way to assure that the
solution fits your needs, in your environment

Don’t panic and have fun

Rash decisions cause serious problems

Know where you are

And make it very obvious!
I.e. color-coded windows & prompts

212

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

SysAdmin Best Practices (pearls)

Keep your propagation constant less than 1. (This comes from nuclear
reactor physics. A reactor with a propagation constant less than 1 is a
generator. More than 1 is a warhead. Basically, don’t let things get out of
control.)
Everything works in front of the salesman.
Don’t cross the streams (Ghostbusters reference — heed safety tips).
If at first you don’t succeed, blame the compiler.
If you finish a project early, the scope will change to render your work
meaningless before the due date.
If someone is trying to save your life, cooperate.
Never beam down to the planet while wearing a red shirt (Star Trek reference
— don’t go looking for trouble).
Learning from your mistakes is good. Learning from someone else’s mistakes
is better.
The fact that something should have worked does not change the fact that it
didn’t.

213

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

SysAdmin Best Practices

The customer isn’t always right, but he pays the bills.
Flattery is flattery, but chocolate gets results.
When dealing on an enigmatic symptom, whether it’s an obscure application
or database error, or a system “hanging”: the Hardware is always guilty until
proven innocent.
Use only standard cross-platform file formats, to share documentation (i.e.,
ASCII files, HTML, or PDF).
Use a log file in every computer to log every change you make.
Share your knowledge and keep no secrets.
Don’t reinvent the wheel, but be creative.
If you can’t live without it, print it out on hardcopy.
Always know where your software licenses are.
Always know where your installation CDs/DVDs/tapes are.
The question you ask as a sys admin is not “Are you paranoid?”; it’s “Are you
paranoid enough?”

214

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

SysAdmin Best Practices

Reboots are for pansies - avoid them at all costs - even when you
think you need to perform one!
Users will eventually find out about the changes you have made to
the system - there is no need to "inform" them with emails, meetings,
man pages, etc.
If you haven't moved the cables - they are not the problem!
Cut your time estimates in half - a good Sys. Admin thrives on
intense situations.
There is no better time to make a change than Friday afternoon,
people will be more than willing to stay a little while extra to help you
test and debug if it is necessary.
The people who write software don't know what they are doing - you
have to chose your own settings every time you install a package
Backups take too long to produce and are rarely needed - make the
system change and "wing it“!

215

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References

You Are Now Free to Move About
Solaris

216

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References

[Kozierok] TCP/IP Guide, No Starch Press, 2005

[Nemeth] Nemeth et al, Unix System Administration
Handbook, 3rd edition, Prentice Hall, 2001

[SunFlash] The SunFlash announcement mailing list
run by John J. Mclaughlin. News and a whole lot more.
Mail sunflash-info@sun.com

Sun online documents at docs.sun.com

[Kasper] Kasper and McClellan, Automating Solaris
Installations, SunSoft Press, 1995

217

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References (continued)

[O’Reilly] Networking CD Bookshelf, Version 2.0,
O’Reilly 2002

[McDougall] Richard McDougall et al, Resource
Management, Prentice Hall, 1999 (and other
"Blueprint" books)

[Stern] Stern, Eisler, Labiaga, Managing NFS
and NIS, 2nd Edition
, O’Reilly and Associates,
2001

218

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References (continued)

[Garfinkel and Spafford] Simson Garfinkel and
Gene Spafford, Practical Unix & Internet
Security, 3rd Ed, O’Reilly & Associates, Inc,
2003 (Best overall Unix security book)

[McDougall, Mauro, Gregg] McDougall, Mauro,
and Gregg, Solaris Internals and Solaris
Performance and Tools, 2007 (great Solaris
internals, DTrace, mdb books)

219

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References (continued)

Subscribe to the Firewalls mailing list by sending
"subscribe firewalls " to
Majordomo@GreatCircle.COM

USENIX membership and conferences. Contact
USENIX office at (714)588-8649 or office@usenix.org

Sun Support: Sun’s technical bulletins, plus access to
bug database: sunsolve.sun.com

Solaris 2 FAQ by Casper Dik:

ftp://rtfm.mit.edu/pub/usenet-by-group/comp.answers/Solaris2/FAQ

220

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References (continued)

Sun Managers Mailing List FAQ by John
DiMarco:

ftp://ra.mcs.anl.gov/sun-managers/faq

Sun's unsupported tool site (IPV6,
printing)

http://playground.sun.com/

Sunsolve STBs and Infodocs

http://www.sunsolve.com

221

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References (continued)

comp.sys.sun.* FAQ by Rob Montjoy: ftp://
rtfm.mit.edu/pub/usenet-by-group/comp.answers/comp-sys-sun-faq

“Cache File System” White Paper from Sun:

http://www.sun.com/sunsoft/Products/Solaris-whitepapers/Solaris-
whitepapers.html

“File System Organization, The Art of
Automounting” by Sun:

ftp://sunsite.unc.edu/pub/sun-info/white-papers/TheArtofAutomounting-1.4.ps

Solaris 2 Security FAQ by Peter Baer Galvin

http://www.sunworld.com/common/security-faq.html

Secure Unix Programming FAQ by Peter Baer
Galvin

http://www.sunworld.com/swol-08-1998/swol-08-security.html

222

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References (continued)

Firewalls mailing list FAQ:

ftp://rtfm.mit.edu/pub/usenet-by-group/
Comp.answers/firewalls-faq

There are a few Solaris-helping files available via anon

ftp at

ftp://ftp.cs.toronto.edu/pub/darwin/
solaris2

Peter’s Solaris Corner at SysAdmin Magazine

http://www.samag.com/solaris

Marcus and Stern, Blueprints for High Availability, Wiley,
2000

Privilege Bracketing in Solaris 10

http://www.sun.com/blueprints/0406/819-6320.pdf

223

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References (continued)

Peter Baer Galvin's Sysadmin Column (and old Pete's
Wicked World security columns, etc)

http://www.galvin.info
My blog at http://pbgalvin.wordpress.com

Operating Environments: Solaris 8 Operating
Environment Installation and Boot Disk Layout by
Richard Elling

http://www.sun.com/blueprints (March 2000)

Sun’s BigAdmin web site, including Solaris and Solaris
X86 tools and information’

http://www.sun.com/bigadmin

224

Saturday, May 2, 2009

Copyright 2009 Peter Baer Galvin - All Rights Reserved

References (continued)

DTrace

http://users.tpg.com.au/adsln4yb/
dtrace.html
http://www.solarisinternals.com/si/dtrace/
index.php
http://www.sun.com/bigadmin/content/dtrace/

225

Saturday, May 2, 2009

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->