Sun Systems Fault

Analysis Workshop

Student Guide - Vol.1

ST-350 Rev G

D61818GC10
Edition 1.0
2009
D62617
Copyright © 2009, Oracle and/or its affiliates. All rights reserved.

Disclaimer

This document contains proprietary information, is provided under a license agreement containing restrictions on use and
disclosure, and is protected by copyright and other intellectual property laws. You may copy and print this document solely for
your own use in an Oracle training course. The document may not be modified or altered in any way. Except as expressly
permitted in your license agreement or allowed by law, you may not use, share, download, upload, copy, print, display,
perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express
authorization of Oracle.

The information contained in this document is subject to change without notice. If you find any problems in the document,
please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This
document is not warranted to be error-free.

Sun Microsystems, Inc. Disclaimer

This training manual may include references to materials, offerings, or products that were previously offered by Sun
Microsystems, Inc. Certain materials, offerings, services, or products may no longer be offered or provided.Oracle and its
affiliates cannot be held responsible for any such references should they appear in the text provided.

Restricted Rights Notice

If this documentation is delivered to the U.S. Government or anyone using the documentation on behalf of the U.S.
Government, the following notice is applicable:

U.S. GOVERNMENT RIGHTS

The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are
restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract.

Trademark Notice

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective
owners.

AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro
Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used
under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark
licensed through X/Open Company, Ltd.
This page intentionally left blank.
This page intentionally left blank.
Contents
About This Course .................................................................Preface-i
Course Goals............................................................................... Preface-i
Topics Not Covered ................................................................... Preface-iii
How Prepared Are You?............................................................ Preface-iv
Introductions ............................................................................... Preface-v
How to Use Course Materials .................................................... Preface-vi
Conventions .............................................................................. Preface-vii
Typographical Conventions ............................................ Preface-viii
Introducing the Fault Analysis and Diagnosis Methodology .......1-1
Objectives ............................................................................................ 1-1
Relevance.............................................................................................. 1-2
Additional Resources ............................................................................ 1-3
Describing the Fault Analysis Methodology ........................................ 1-4
Two–Stage Methodology............................................................. 1-4
Clearly State the Problem ............................................................ 1-5
List the Facts ................................................................................ 1-6
Documenting Each Item Carefully ............................................ 1-12
Introducing the Fault Diagnosis Methodology ................................... 1-15
Prioritizing Planned Tests .......................................................... 1-16
Verifying the Corrective Action ................................................ 1-18
Documenting Each Item ............................................................ 1-18
Identifying the Basic Layers and Error Types in Sun Systems .......... 1-21
Overview of the Four Basic Layers of a Sun System ................ 1-21
Introducing Sun System Fault Types......................................... 1-23
Identifying Error–Reporting Mechanisms ................................. 1-26
Exercise: Performing Fault Analysis and Diagnosis .......................... 1-30
Preparation ................................................................................. 1-30
Tasks .......................................................................................... 1-30
Fault Analysis and Diagnosis Worksheet Template .......................... 1-32
Analysis Phase ........................................................................... 1-32
Diagnosis Phase ......................................................................... 1-33
Exercise Summary .............................................................................. 1-34
Exercise Solution ................................................................................ 1-35

vii
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G
 Performing Solaris OS Diagnostics ............................................... 2-1
Objectives ............................................................................................ 2-1
Relevance.............................................................................................. 2-2
Additional Resources ............................................................................ 2-3
Using the CPU and Memory Management Commands........................ 2-4
Using the ps Command ............................................................... 2-4
The prstat Command ............................................................... 2-6
The proc Tools ........................................................................... 2-8
Using the vmstat Command .................................................... 2-10
Using the psrinfo Command .................................................. 2-12
Using the mpstat Command .................................................... 2-13
Using the pgrep Command ...................................................... 2-15
Using the modinfo Command .................................................. 2-15
Using the Program Execution Management Commands .................... 2-17
Using the truss Command ...................................................... 2-17
Introduction to Crash Dump and Core Dump............................ 2-19
Using the coreadm Command .................................................. 2-20
Using the General–Purpose Commands ............................................. 2-22
Using the find Command......................................................... 2-22
Using the strings Command....................................................... 2-22
Using the syslogd Command ..................................................... 2-23
Using the dmesg Command ...................................................... 2-23
Using the script Command .................................................... 2-24
Using the file Command......................................................... 2-25
Using the tail Command......................................................... 2-26
Using the uname Command ...................................................... 2-26
Using the prtconf Command .................................................. 2-28
Using the sysdef Command .................................................... 2-30
Using the nm Command ............................................................. 2-31
Using the swap Command......................................................... 2-31
Using the Software Package Management Commands ...................... 2-33
Using the pkgchk Command .................................................... 2-33
Using the pkginfo Command .................................................. 2-35
Using the pkgadd Command .................................................... 2-35
Using the pkgparam Command................................................ 2-36
Using the pkgrm Command ...................................................... 2-38
Using the showrev Command .................................................. 2-38
Using the File–Checking Commands ................................................. 2-40
Checking for Hidden Characters................................................ 2-40
Comparing File Contents ........................................................... 2-43
Using the Network Management Commands ..................................... 2-45
Secure by Default Installation.................................................... 2-45
Using the ping Command......................................................... 2-46
Using the traceroute Command ........................................... 2-47
Using the ifconfig Command................................................ 2-49
Using the arp Command........................................................... 2-51

viii Sun System Fault Avalysis Workshop

Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G
 Using the route Command ...................................................... 2-53
Using the netstat Command .................................................. 2-54
Using the snoop Command ...................................................... 2-56
Using the netservices Command................................................ 2-57
Using the inetadm Command .................................................. 2-58
Using the Device Management Commands........................................ 2-60
Using the devfsadm Command................................................ 2-60
Using the Disk and File System Management Commands................. 2-62
Using the format Command .................................................... 2-62
Using the fdisk Command ...................................................... 2-65
Using the fsck Command......................................................... 2-67
Using the fmthard Command .................................................. 2-70
Using the prtvtoc Command .................................................. 2-71
Using the fstyp Command ...................................................... 2-72
Using the iostat Command .................................................... 2-72
Using the Storage Management Commands....................................... 2-76
Using the iscsiadm Command................................................ 2-76
Using the luxadm Command .................................................... 2-77
Using the cfgadm Command .................................................... 2-79
Exercise: Performing Solaris OS Diagnostics .................................... 2-81
Preparation ................................................................................. 2-81
Tasks .......................................................................................... 2-81
Exercise Summary .............................................................................. 2-83
Exercise Solutions............................................................................... 2-84
Introducing OBP Components, Features, and Diagnostics .........3-1
Objectives ............................................................................................ 3-1
Relevance.............................................................................................. 3-2
Additional Resources ............................................................................ 3-3
Introducing OBP Components.............................................................. 3-4
Introducing Boot PROM.............................................................. 3-7
Introducing Non–Volatile RAM ......................................................... 3-10
Storing Custom Values in the nvramrc Variable ..................... 3-10
Listing Common OBP Variables ............................................... 3-11
Modifying OBP Variables and Running Diagnostics......................... 3-13
Using the Power–On Commands............................................... 3-13
Listing Common OBP Variables ............................................... 3-13
Modifying OBP Variables ......................................................... 3-15
Using Manual OBP Diagnostic Commands .............................. 3-18
Using OBP Commands to Display System Information ........... 3-21
Exercise: Modifying the OBP Variables ............................................ 3-23
Preparation ................................................................................. 3-23
Tasks .......................................................................................... 3-23
Exercise Solutions............................................................................... 3-24
Exercise: Performing Manual OBP Diagnostics ................................ 3-25
Preparation ................................................................................. 3-25

ix
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G
 Tasks ......................................................................................... 3-26
Exercise Summary .............................................................................. 3-27
Exercise Solutions............................................................................... 3-28
Enabling and Monitoring POST Diagnostics................................. 4-1
Objectives ............................................................................................ 4-1
Relevance.............................................................................................. 4-2
Additional Resources ............................................................................ 4-3
Introducing POST Concepts ................................................................. 4-4
Identifying the Testable Components .......................................... 4-4
Describing the diag–switch? Variable.................................... 4-5
Enable Extended POST Diagnostics............................................ 4-6
Manipulating the OBP Device Tree.................................................... 4-10
Device Path Name ..................................................................... 4-13
Automated OBP Probing ........................................................... 4-15
Navigating and Examining the OBP Device Tree ..................... 4-17
Creating Custom Device Aliases ............................................... 4-20
Booting From the diag–device or boot–device Variable.. 4-25
Viewing Extended Diagnostics During POST.................................... 4-27
Using the tip Command........................................................... 4-27
Using the prtdiag Command .................................................. 4-32
Using the show–post–results Command ............................ 4-40
Using the forthdebug command ............................................ 4-41
Exercise: Enabling and Monitoring POST Diagnostics ..................... 4-42
Preparation ................................................................................. 4-42
Tasks ......................................................................................... 4-43
Exercise Summary .............................................................................. 4-45
Exercise Solutions............................................................................... 4-46
The Boot Sequence ......................................................................... 5-1
Objectives ............................................................................................ 5-1
Relevance.............................................................................................. 5-2
Additional Resources ............................................................................ 5-3
Compare Boot Process Phases .............................................................. 5-4
Boot PROM Phase ....................................................................... 5-5
SPARC Boot Programs Phase ..................................................... 5-7
GRUB Introduction...................................................................... 5-7
GRUB-Based Booting ................................................................. 5-8
The GRUB Menu....................................................................... 5-11
Using the bootadm Command to Manage GRUB.................... 5-12
Editing the /boot/grub/menu.lst File................................ 5-13
Modifying x86 System Boot Behavior in the Solaris OS .......... 5-13
Setting Solaris Boot Parameters Using the eeprom Command 5-14
x86/x64 Boot PROM Phase ....................................................... 5-16
x86/x64 Boot Programs Phase ................................................... 5-17
The kernel Initialization Phase ............................................. 5-18
The /etc/system File and Kernel Configuration................... 5-21

x Sun System Fault Avalysis Workshop

Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G
 The init Phase ........................................................................ 5-25
Run Level Fundamentals .......................................................... 5-27
Changing Run Levels................................................................. 5-29
Controlling Legacy Boot Processes .......................................... 5-30
The /sbin Directory ................................................................. 5-30
The /etc/rc#.d Directories .................................................. 5-32
The svc.startd Daemon .............................................................. 5-33
Identifying Common Errors in a Boot Sequence................................ 5-35
Booting From Incorrect Device Type ........................................ 5-35
Booting From Incorrect Disk ..................................................... 5-35
Recovering Corrupted Solaris Image......................................... 5-36
Using boot Commands............................................................. 5-39
Exercise: Introducing the OBP Device Tree and the Boot Sequence. 5-42
Preparation ................................................................................. 5-42
Tasks .......................................................................................... 5-42
Exercise Summary .............................................................................. 5-44
Exercise Solutions............................................................................... 5-45
Predictive Self–Healing....................................................................6-1
Objectives ............................................................................................ 6-1
Relevance.............................................................................................. 6-2
Additional Resources ............................................................................ 6-3
Describe ctfs and objfs Filesystems................................................ 6-4
The Contract File System ............................................................ 6-4
Kernel Object File System ........................................................... 6-5
Describe the Benefits of Predictive Self–Healing ................................ 6-6
The Fault Manager....................................................................... 6-6
The Service Manager ................................................................... 6-6
Describe the Fault Management Architecture ...................................... 6-8
Features ........................................................................................ 6-8
Event Naming Schemes ............................................................. 6-10
Interpret Recommended Actions ........................................................ 6-14
Looking at FMA Data................................................................ 6-14
Describe the Service Management Facility ........................................ 6-25
Features ...................................................................................... 6-25
The SMF Architecture ............................................................... 6-26
Services ...................................................................................... 6-27
Managing Services Using SMF .......................................................... 6-38
Services Daemons...................................................................... 6-38
Command–line Tools................................................................. 6-39
Diagnosing Problems Using SMF ...................................................... 6-50
Debugging a Hang on Boot ....................................................... 6-50
Using Debug Mode .................................................................... 6-52
Debugging a Service .................................................................. 6-54
Repository Problems.................................................................. 6-55
Introducing Types of System Failures ...........................................7-1

xi
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G
 Objectives ............................................................................................ 7-1
Relevance.............................................................................................. 7-2
Additional Resources ............................................................................ 7-3
Introducing System Crashes and Core Files ......................................... 7-4
Differentiating Crash Dumps from Core Files ............................ 7-4
Introducing System Panics........................................................... 7-6
System Hangs .............................................................................. 7-9
Generating a System Crash Dump...................................................... 7-11
Writing the System Crash Dump ............................................... 7-11
Using the savecore Command Automatically........................ 7-14
Using the savecore Command Manually ............................... 7-14
Managing Application Core Dumps .......................................... 7-16
Interpret Watchdog Resets.................................................................. 7-18
Identifying Causes and Effects of Watchdog Resets ................. 7-18
Identifying the watchdog–reboot? OBP Variable ................ 7-19
Displaying the Register Contents by Using OBP Commands... 7-19
Identifying the misc/obpsym Kernel Module ......................... 7-19
Exercise: Introducing Types of System Failures ................................ 7-21
Preparation ................................................................................. 7-21
Tasks .......................................................................................... 7-21
Exercise Summary .............................................................................. 7-22
Exercise Solutions............................................................................... 7-23
Analyzing Core Dumps Using the mdb Utility ................................ 8-1
Objectives ............................................................................................ 8-1
Relevance.............................................................................................. 8-2
Additional Resources ............................................................................ 8-3
Introducing the mdb Utility................................................................... 8-4
The mdb Utility ............................................................................ 8-5
Launching mdb............................................................................. 8-6
Getting mdb Help........................................................................ 8-7
Features of the mdb Utility........................................................... 8-9
General mdb Command Formats ............................................... 8-11
Analyze the Source of a Crash Dump................................................. 8-13
Identifying Macros and Registers .............................................. 8-13
Examining System Dumps Using the mdb Utility ..................... 8-16
Exercise: Analyzing Core Dumps Using the mdb Utility ................... 8-27
Preparation ................................................................................. 8-27
Tasks .......................................................................................... 8-27
Exercise Summary .............................................................................. 8-29
Exercise Solutions............................................................................... 8-30
Dynamic Tracing Facility................................................................. 9-1
Objectives ............................................................................................ 9-1
Relevance.............................................................................................. 9-2
Additional Resources ............................................................................ 9-3
Introduce The Solaris Dynamic Tracing Facility ................................. 9-4

xii Sun System Fault Avalysis Workshop

Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G
 DTrace Features ........................................................................... 9-4
DTrace Architecture .................................................................... 9-5
Interpret DTrace Command–Lines ..................................................... 9-11
Using the dtrace Command .................................................... 9-11
Specifying Probes in DTrace ..................................................... 9-13
Interpreting Probes..................................................................... 9-13
DTrace Actions .......................................................................... 9-17
Action Examples........................................................................ 9-18
Selecting DTrace Toolkit Tools.......................................................... 9-23
The Scripts ................................................................................. 9-23
The man Pages ........................................................................... 9-24
The Example Files ..................................................................... 9-24
The oneliners File ................................................................. 9-25
Writing D Scripts ................................................................................ 9-26
Executable D Scripts.................................................................. 9-26
Diagnosing Faults Using Online Tools.......................................... A-1
Additional Resources ........................................................................... A-2
Using the Online Man Pages ............................................................... A-3
The MANPATH Variable ............................................................... A-3
The man Command ..................................................................... A-4
Diagnosing Problems by Using the SunSolve Online Service ............ A-8
Accessing the SunSolve Online Service ..................................... A-8
Using the SunSolve Online Service........................................... A-9
Performing Search Operations in SunSolve Online Service ... A-14
Viewing the Patch Support Portal............................................ A-18
Using the Sun Explorer Data Collector Utility................................. A-22
Watch the Video ....................................................................... A-22
Obtaining Explorer ................................................................... A-22
Installing Explorer .................................................................... A-23
Configuring and Executing Explorer ........................................ A-24
Reviewing the Explorer Output ................................................ A-25
Using the docs.sun.com Web Site................................................ A-26
Browsing the docs.sun.com Web Site.................................. A-27
Printing Files From the docs.sun.com Web Site ................. A-30
Sample Outputs ............................................................................... B-1
Output of the eeprom Command on a Sun4U Enterprise Server ........B-2
Sample Report of The PatchDiag Tool..............................................B-4
Workshop Exercises ....................................................................... C-1
Introduction.......................................................................................... C-1
Preparatory Tasks ........................................................................C-2
Fault 1 – Users Can’t Change Their Passwords ...................................C-3
Fault 2 – System Doesn’t Boot .............................................................C-5
Fault 3 – ps (1) No Longer Works ....................................................C-7
Fault 4 – Repetitive Boot Sequences ....................................................C-9

xiii
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G
 Fault 5 – Student Can’t Login.............................................................C-11
Fault 6 – Problem With the root Login ............................................C-13
Fault 7 – Network Problems ...............................................................C-15
Fault 8 – Boot Into Maintenance Mode ..............................................C-17
Fault 9 – The ps Command is Broken................................................C-19
Fault 10 – Slow ftp Performance ......................................................C-21
Fault 11 – Users Can’t Login to Desktop ...........................................C-23
Fault 12 – Network Problem...............................................................C-25
Fault 13 – Can’t Login as Root...........................................................C-27
Fault 14 – Can’t Login to Desktop .....................................................C-29
Fault 15 – Root Can’t Get In ..............................................................C-31
Fault 16 – Boot Failure .......................................................................C-33
Fault 17 – X-Server Doesn’t Start ......................................................C-35
Fault 18 – System Hangs During Boot ...............................................C-37
Fault 19 – Problem With Read-Only File System ..............................C-39
Fault 20 – Root Can’t Login ...............................................................C-41
Fault 21 – Network Problem...............................................................C-43
Fault 22 – Network Problem...............................................................C-45
Fault 23 – Can’t Read man Pages .......................................................C-47
Fault 24 – System Makes Inappropriate Sound at Boot .....................C-49
Fault 25 – System Doesn’t Boot .........................................................C-51
Fault 26 – Primary Network Interface is Gone...................................C-53
Fault 27 – System Is Hung..................................................................C-55
Fault 28 – Inappropriate Halts ............................................................C-57
Fault 29 – Commands Don’t Execute.................................................C-59
Fault 30 – System Doesn’t Boot .........................................................C-61
Fault 31 – System Doesn’t Boot .........................................................C-63
Fault 32 – System Seems Dead...........................................................C-65
Fault 33 – Login Problem ...................................................................C-67
Fault 34 – System Crash Dump ..........................................................C-69
Fault 35 – Slow Zone..........................................................................C-71
Fault 36 – Using vi Generates a Core File.........................................C-73
Fault 37 – Page (pg) Doesn’t Work....................................................C-75
Fault 38 – Repository Problem ...........................................................C-77
Fault 39 – vi Not Working.................................................................C-79
Fault 40 – Network Problem...............................................................C-81
Fault 41 – System Powers Off at Random Times...............................C-83
Fault 42 – Login Doesn't Allow Login ...............................................C-85
Fault 43 – Network Problem...............................................................C-87
Fault 44 – Server Not Responding to Network Traffic ......................C-89
Fault 45 – Network Problem...............................................................C-91
Fault 46 – Time is Slow......................................................................C-93
Fault 47 – Random Date Displays ......................................................C-95
Fault 48 – NFS Doesn’t Work on Server............................................C-97
Fault 49 – System Seems Slow...........................................................C-99
Fault 50 – Users Don't Have Homes.................................................C-101

xiv Sun System Fault Avalysis Workshop

Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G
Workshop Exercise Solutions........................................................ D-1
Fault 1 – Users Can’t Change Their Passwords .................................. D-3
Fault 2 – System Doesn’t Boot ............................................................ D-4
Fault 3 – ps(1) No Longer Works ..................................................... D-5
Fault 4 – Repetitive Boot Sequences ................................................... D-6
Fault 5 – Student Can’t Login.............................................................. D-7
Fault 6 – Problem With the root Login ............................................. D-8
Fault 7 – Network Problem.................................................................. D-9
Fault 8 – Boot Into Maintenance Mode ............................................. D-10
Fault 9 – The ps Command is Broken............................................... D-11
Fault 10 – Slow ftp Performance ..................................................... D-12
Fault 11 – Users Can’t Login to Desktop .......................................... D-13
Fault 12 – Network Problem.............................................................. D-14
Fault 13 – Can’t Login as Root.......................................................... D-15
Fault 14 – Can’t Login to Desktop .................................................... D-16
Fault 15 – Root Can't Get In .............................................................. D-17
Fault 16 – Boot Failure ...................................................................... D-18
Fault 17 – X-Server Doesn’t Start ..................................................... D-19
Fault 18 – System Hangs During Boot .............................................. D-20
Fault 19 – Problem With Read-Only File System ............................. D-21
Fault 20 – Root Can’t Login .............................................................. D-23
Fault 21 – Network Problem.............................................................. D-24
Fault 22 – Network Problem.............................................................. D-25
Fault 23 – Can’t Read man Pages ...................................................... D-26
Fault 24 – System Makes Inappropriate Sound at Boot .................... D-27
Fault 25 – System Doesn’t Boot ........................................................ D-31
Fault 26 – Primary Network Interface is Gone.................................. D-32
Fault 27 – System Is Hung................................................................. D-34
Fault 28 – Inappropriate Halts ........................................................... D-38
Fault 29 – Commands Don’t Execute................................................ D-39
Fault 30 – System Doesn’t Boot ........................................................ D-40
Fault 31 – System Doesn’t Boot ........................................................ D-41
Fault 32 – System Seems Dead.......................................................... D-42
Fault 33 – Login Problem .................................................................. D-43
Fault 34 – System Crash Dump ......................................................... D-44
Fault 35 – Slow Zone......................................................................... D-50
Fault 36 – Using vi Generates a Core File........................................ D-52
Fault 37 – Page (pg) Doesn’t Work................................................... D-53
Fault 38 – Repository Problem .......................................................... D-54
Fault 39 – vi Not Working................................................................ D-55
Fault 40 – Network Problem.............................................................. D-56
Fault 41 – System Powers Off at Random Times.............................. D-57
Fault 42 – Login Doesn't Allow Login .............................................. D-58
Fault 43 – Network Problem.............................................................. D-59
Fault 44 – Server Not Responding to Network Traffic ..................... D-60
Fault 45 – Network Problem.............................................................. D-61

xv
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G
 Fault 46 – Time is Slow..................................................................... D-62
Fault 47 – Random Date Displays ..................................................... D-64
Fault 48 – NFS Doesn’t Work on Server........................................... D-66
Fault 49 – System Seems Slow.......................................................... D-67
Fault 50 – Users Don't Have Homes.................................................. D-70
Sun Integrated Lights-Out Manager Overview.............................. E-1
Table of Contents..................................................................................E-3
Introduction...........................................................................................E-4
What Is ILOM? ............................................................................E-4
What Does ILOM Do?................................................................ E-5
ILOM Features and Functionality................................................E-5
ILOM Interfaces ..........................................................................E-7
Platform Support for ILOM .........................................................E-7
Getting Started With ILOM .........................................................E-8
.For More Information .................................................................E-9
Additional Information..................................................................... F-1
The probe Commands ......................................................................... F-2
The test Commands ........................................................................... F-4
The watch Commands ......................................................................... F-6
Introducing FPROM Upgrades ............................................................. F-7
The show–post–results Commands............................................... F-9

xvi Sun System Fault Avalysis Workshop

Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision G