Troubleshooting/Maintenance of HP-UX

Where to get information about problems:

• dmesg --> provides a finite list of diagnostic messages

• /var/adm/syslog/syslog.log --> system log
• /opt/resmon/log/error.log -->
• /etc/shutdownlog --> shutdown information
• /etc/rc.log --> system startup log
• /var/tombstones/ts99 --> crash analysis file
• cstm - command line support tool manager
• mstm - menu based support tool manager
o <alt><underlined letter of command>
o <tab> --> to move to another portion of the screen, such as the drop
down menu area
• Service Processor
o <ctrl> <b> from a serial console
o he - help
o co - return to console mode (exits the program)
o sl - show log

Panic Reboots

Check these files for clues:

• /var/tombstones/ts99
• /etc/shutdownlog

Bad disk

1. Check the syslog (/var/adm/syslog/syslog.log) looking for disk errors.

2. Check the ioscan (ioscan -fnC disk), looking for NO_HW rather than Claimed.
3. If diaglogd is running then check STM logs (/var/opt/resmon/log/event.log)
4. Check the volume group to see if the disk is listed and whether there is any problem
with it's status (vgdisplay -v | more)
5. Check lvmtab to see if the disk is supposed to be in a volume group (strings
/etc/lvmtab | more)

Filesystem do not mount after a reboot

1. Reactivate the Volume Group --> vgchange -a y /dev/<volume group>

2. Remount the filesystems --> mount -a
3. If still no success then perform a filesystem check --> fsck
/dev/<volumegroup>/<logicalvolume>
4. Remount the filesystems --> mount -a
5. Check to see if all the filesystems are there:
a) bdf
b) compare with /etc/fstab

Filesystem full

du -kx / | sort -rn | more

du -akx | sort -nr | more

Shows directories on the local filesystem and how much space they are taking up

NFS mount - Permission Denied

1. Check to see if the format of the /etc/exports file is correct on the server that is the nfs
server.

2. exportfs -av to export the filesystem

3. Check the /etc/fstab file on the client to make sure that it is correct

4. /usr/sbin/showmount -e <server> on the client to show what is being exported

5. To bypass the /etc/exports file execute the following on the nfs server: exportfs -i -o
rw <filesystem>.

NFS Server

/etc/rc.config.d/nfsconf --> NFS_SERVER=1

Verify the proper processes are running:

/sbin/init.d/nfs.server stop

The processes should NOT be running:

# ps -ef|grep nfsd
# ps -ef|grep rpc.mountd
# ps -ef|grep rpc.lockd
# ps -ef|grep rpc.statd

/sbin/init.d/nfs.server start
These processes should be running:

# ps -ef|grep nfsd
root 3444 1 0 10:39:12 ?
root 3451 3444 0 10:39:12 ?
root 3449 3444 0 10:39:12 ?
root 3445 3444 0 10:39:12 ?
# ps -ef|grep rpc.mountd
root 3485 1 0 10:42:09 ?
# ps -ef|grep rpc.lockd
root 3459 1 0 10:39:12 ?
# ps -ef|grep rpc.statd
root 3453 1 0 10:39:12 ?
0:00 /usr/sbin/nfsd 4
0:00 /usr/sbin/nfsd 4
0:00 /usr/sbin/nfsd 4
0:00 /usr/sbin/nfsd 4
0:00 rpc.mountd
0:00 /usr/sbin/rpc.lockd
0:00 /usr/sbin/rpc.statd

To start a process if it is not running:

# ps -ef|grep rpc.mountd
# rpc.mountd or /usr/sbin/rpc.mountd
# ps -ef|grep rpc.mountd
root 3485 1 0 10:42:09 ? 0:00 rpc.mountd

/etc/inetd.conf needs to have the proper services active (not commented out)

##
# WARNING: The rpc.mountd should now be started from a startup script.
# Please enable the mountd startup script to start rpc.mountd.
##
#rpc stream tcp nowait root /usr/sbin/rpc.rexd 100017 1 rpc.rexd
rpc dgram udp wait root /usr/lib/netsvc/rstat/rpc.rstatd 100001 2-4 rpc.rstatd
rpc dgram udp wait root /usr/lib/netsvc/rusers/rpc.rusersd 100002 1-2
rpc.rusersd
rpc dgram udp wait root /usr/etc/rpc.mountd 100005 1 rpc.mountd -e
rpc dgram udp wait root /usr/lib/netsvc/rwall/rpc.rwalld 100008 1
rpc.rwalld
#rpc dgram udp wait root /usr/sbin/rpc.rquotad 100011 1 rpc.rquotad
rpc dgram udp wait root /usr/lib/netsvc/spray/rpc.sprayd 100012 1
rpc.sprayd

NIC problems:

The lanadmin utility provides NIC statistics

The nettladmin utility provides packet trace information
Replacing a Mirrored Root Disk:

• Replace the disk

o Hot swap can be performed while system is up
o Not hot swappable means the system must be brought down
• Reboot the system into single user mode
o shutdown -r 0, unless the system is powered off already, then power it
back on
o interrupt the boot
o bo pri (or bo alt if the disk that was replaced was the primary boot disk)
o IPL>hpux -is -lq (;0)/stand/vmunix
• vgcfgrestore -n /dev/vg00 /dev/rdsk/c?t?d?
• vgsync /dev/vg00
• mkboot /dev/rdsk/c?t?d?
• mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c?t?d?
• shutdown -r 0
• lvlnboot -v /dev/vg00 to verify that the disk is seen as bootable

Software Installation (swinstall, sd, etc)

ERROR: "server::/tmp/omni_tmp/packet": You do not have the

required permissions to perform this operation. Check
permissions using the "swacl" command or see your system
administrator for assistance. Or, to manage applications
designed and packaged for nonprivileged mode, see the
"run_as_superuser" option in the "sd" man page.
WARNING: More information may be found in the daemon logfile on this
target (default location is
server:/var/adm/sw/swagentd.log).

Bounce swagentd daemon: /usr/sbin/swagentd -r