Monitoring AIX Users

Audience: AIX Administrators and End Users

Date: July 16, 1999
Monitoring user activity is an important system administration task. Here are a few useful
AIX commands for monitoring user activity.
Command Description
Display information about previous logins, including login
last
date/time, logout time, originating remote host.
who List users who are currently logged on.
ps -fu "userid" List processes being run by "userid".
Text file located in the home directory of each Korn shell user.
.sh_history
The file contains the last 50 commands issued by the user.
/var/adm/sulog Text file containing all "su" activity.
Binary file containing a listing of all failed login attempts. The
/etc/security/failedlogin
file can be read by only root, using "who failedlogin"
AIX has two other facilities for monitoring user activity on a more granular level. The
first is "Accounting" which monitors user's connect time and system usage (cpu by
command, diskstorage and printer usage). The second is "Auditing", a security related
function, which provides a detailed audit trail of each user's activity, including priviledge
failures, commands run, files they view/create/delete, and more. Both Accounting and
Auditing are part of the base AIX operating system. However, both must be configured to
run. The attachment HTML files discuss how to set up accounting and auditing. See your
AIX documentation for more information.

AIX System Security Audit [audit.all.cmd]

AIX System Security Audit

-----------------------------------------------------------------------
--------

Contents

About This Document

Related Documentation
Overview
Auditing Events and Objects
Auditing Mode: BIN and STREAM
Starting and Stopping Audit
Auditing Configuration
Auditing a User
Auditing an Object
Disk Space Consideration
Understanding the Output
Common Problems with Auditing
Data Overload
Fixes for Auditing Subsystem
-----------------------------------------------------------------------
--------

About This Document

This document is intended to simplify the use of the auditing system

provided
in AIX and applies to all versions of AIX. It includes information on
what
auditing offers, what its requirements are, and what common problems
may be
encountered. The intention is not to answer every question about
auditing, but
to provide a starting point for understanding and setting up auditing.

Related Documentation

Documentation for System Auditing can be found in Chapter 5 of the

"System
Management Guide" for AIX version 3.2, Chapter 3 of the "System
Management
Guide: Operating System and Devices" for AIX version 4, and Chapter 8
in the
IBM Redbook "Elements of Security: AIX 4.1" (GG24-4433-00).

The AIX and RS/6000 product documentation library is also available:

http://www.rs6000.ibm.com/resource/
-----------------------------------------------------------------------
--------

Overview

The auditing subsystem provides the means to record security-related

information and to alert system administrators of potential and actual
violations of the system security policy. The information collected by
auditing
includes: the name of the auditable event, the status (success or
failure) of
the event, and any additional event-specific information related to
security
auditing.
-----------------------------------------------------------------------
--------

Auditing Events and Objects

A list of audit events built into AIX, along with a list of predefined
audit
objects, can be found in the file /etc/security/audit/events.

In general, auditing events are defined at the system call level. A

single
operation at the command line would result in records of several events
in the
audit trail. For example, when viewing a file using the cat or more
command,
you would see the following records logged into the audit trail:

FILE_Open (file is opened)

FILE_Read (file is read)
FILE_Write (file is written to standard output)
PROC_Create (process creation for more OR cat)
PROC_Execute (command execution)
PROC_Delete (process completion)

Auditing all possible events can produce a huge amount of data. Through
audit
controls (that is, modifying the configuration files), you can select
events to
be recorded.

Audit events are grouped into classes. The events can be defined by
which
events are in a class. While the class names are arbitrary, they,
rather than
individual event names, are associated with user IDs when the audit
subsystem
is active.

Auditing objects are just individual files that will be monitored. Three
operations can be audited: read, write, and execute. Objects are not
associated
with user IDs. Audit records are generated whenever an audited object is
referenced by any user (including root).

To add further audit objects, extend the /etc/security/audit/objects

file.
-----------------------------------------------------------------------
--------

Auditing Mode: BIN and STREAM

There are two modes of operation for auditing: BIN and STREAM. BIN mode
writes
the audit trail to alternating temporary files (bins), then to a single
trail
file. STREAM mode writes to a circular buffer that is read synchronously
through an audit pseudo-device (/dev/audit).

An audit can be started in one OR both of these modes.

Using the audit configuration setup shipped with AIX,

/etc/security/audit/config, the BIN mode alternates between /audit/bin1
and
/audit/bin2. When one BIN is full (the binsize parameter determines the
size of
the bin), the audit switches to the other BIN file while adding the
accumulated
data in the first file to the audit trail (defined in
/etc/security/audit/bincmds), /audit/trail. Use "audit shutdown" to be
certain
that all audit records have been added to /audit/trail. The BIN mode
audit
record is in binary format. You can read it with audit commands such as
auditpr.

In STREAM mode, the default AIX configuration provides a program to

read the
STREAM buffer and processes each record with the commands found in
/etc/security/audit/streamcmds. These commands format the output into
human-readable form and write it in /audit/stream.out. This file is NOT
cumulative; it is restarted every time the audit is restarted. The
STREAM audit
trail can be read in real time by reading /audit/stream.out, or by
sending
output directly to a terminal or printer.
-----------------------------------------------------------------------
--------

Starting and Stopping Audit

There are five audit subcommands for invoking auditing. They are as
follows:

audit start - to activate the audit subsystem (This is

the only correct way to start audit.)
audit shutdown - to stop auditing subsystem, processing
final BIN records and removing the
/audit/auditb file that is used as an
"active" indicator by the audit modules
audit off - to suspend auditing temporarily
audit on - to resume audit after audit off
audit query - to display the status of auditing

NOTE: Using audit commands in the wrong order can confuse the auditing
subsystem. If the auditing subsystem gets confused, reset everything by
deleting all files in the /audit directory (except trail, stream.out
and bin
files).

Auditing can be run at the discretion of the system administrator.

Depending on
the environment, it is usually not necessary to have auditing running
at all
times. If it is configured to monitor a large number of events or
objects at
all times, the amount of data generated would be so substantial that its
overhead would outweigh its benefit. It is worthwhile to take time to
configure
auditing to collect selected information.

To start auditing at system startup, add the following line to the

/etc/rc
file, just prior to the line reading dspmsg rc.cat 5 'Multi-user
initialization
completed':

/usr/sbin/audit start
If auditing is running at all times, make sure to run or to add the
following
in the /usr/sbin/shutdown script to properly shut down:

/usr/sbin/audit shutdown

-----------------------------------------------------------------------
--------

Auditing Configuration

All auditing related configuration files reside in /etc/security/audit.

The
/etc/security/audit/config file contains the key audit controls. It has
the
following stanzas:

start - specifies whether BIN or STREAM

(or both) should be used for auditing
bin and stream - contain controls for each mode; the names
of the BIN files are specified here
classes - defines several groups (classes) of
auditing events

The predefined classes are: general, objects, SRC, kernel, files,

SVIPC, mail,
cron, and TCPIP. New classes can be defined using the auditing events
in the
/etc/security/audit/events file. All audit classes except the objects
class are
associated with user IDs. For example, audit the events defined as
"general"
and "TCPIP" for user root.

user stanza - lists specified users and the audit classes

assigned to them; each username must be the
login name of a system user or the string
'default'

An example of this stanza is as follows:

users:
root = general
joe = general, files, TCPIP
default = general

When auditing starts, it ALWAYS audits the events specified for every
user ID
defined in the config file and ALL the objects defined in
/etc/security/audit/objects. If the objects' audit records are not
wanted,
remove or comment out (using an *) the objects defined in the objects
file.

If there are specific classes of events that are not wanted as audit
records,
specify No_Events for that specific class in the config file.

For example:

files = No_Events
or
tcpip = No_Events

The objects file contains all objects to be audited when auditing is

active. A
user defined object is displayed as:

/home/joe/my.stuff:
r = "JOE_READ"
w = "JOE_WRITE"

The names JOE_READ and JOE_WRITE are referenced in the

/etc/security/events
file to define the format of the auditpr output:

JOE_READ = printf "%s"

JOE_WRITE = printf "%s"

NOTE: There is no need to add the newly added objects to the objects
stanza in
the /etc/security/audit/config file, since the objects line is not
referenced.
Only the objects file is referenced.

The streamcmds file has commands that are entered for STREAM audit
records. The
default file contains one command.

Enter:

/usr/sbin/auditstream | auditpr > /audit/stream.out &

Adding the -v flag for the auditpr command improves this command at the
expense
of having more information. Without -v, full path names for files are
not shown
in the audit output; only file descriptors are recorded.

To limit the amount of data collected during the auditing operation,

use the -c
option on the auditstream command to select a specific class of events
as
defined in the config file, or use the auditselect command to select
specific
events.

For example:

NOTE: This command must be all on one line in the streamcmds file. This
command
will collect only FILE_Open event records.
 /usr/sbin/auditstream | /usr/sbin/auditselect -e "event == FILE_Open"
| auditpr -v > /audit/stream.out &

NOTE: The following command will limit data collection to only the
TCPIP class
of events as defined in the config file.

/usr/sbin/auditstream -c tcpip | auditpr -v > /audit/stream.out &

The bincmds file contains commands that are entered whenever a BIN file
fills
or when auditing is shut down. The file distributed reads like the
following:

/usr/sbin/auditcat -p -o $trail $bin

The environment variables in the preceding command are defined while

auditing
is active. The auditselect command can be added to select specific
events,
reducing the amount of audit records.

The bincmds file will only collect audit records that match USER_SU or
USER_Login audit events. Enter:

/usr/sbin/auditselect -e "event== USER_SU || event== \

USER_Login" $bin > /audit/trail.login

-----------------------------------------------------------------------
--------

Auditing a User

For example:

1. To audit classes, use the fastpath command smit chuser.

*User NAME [joe]

AUDIT classes [general, files]

A user stanza should be displayed for joe in

/etc/security/audit/config
file.
2. At the command line, edit the /etc/security/config file. In the
classes
stanza, add the following new class:

procmon = PROC_Create, PROC_Delete, PROC_Execute

In the users stanza, the following could exist:

joe = procmon

The newly assigned audit classes will take effect at the next
login for
user joe.
3. The BIN mode audit trail can be read with the following:
 auditpr -v < /audit/trail | more

The STREAM mode audit file /audit/stream.out can be viewed directly.

Remember
that the /audit/stream.out file is rewritten each time the auditing
subsystem
is started. Save the old stream.out before starting auditing.

If you do not want the objects audit records when auditing a user ID,
comment
out the objects defined in the /etc/security/audit/objects file or
rename this
file.
-----------------------------------------------------------------------
--------

Auditing an Object

In the following example, all processes writing to the /etc/utmp file

will be
audited.

1. Edit the /etc/security/audit/objects file to add the following:

/etc/utmp:
w = "UTMP_WRITE"

2. Edit the /etc/security/audit/events file to include the following:

* /etc/utmp
UTMP_WRITE = printf " %s "

The audit record is displayed as follows:

UTMP_WRITE root OK Wed Jul 12 12:12:25 1995 init

In this case, the init process owned by root wrote to the file.

NOTE: The length of an audit event or object name cannot exceed 15

characters.
This limit is defined in the header file /usr/include/sys/audit.h,
ah_event
[16]. The following error message usually indicates an invalid event or
object
name.

"auditevents (): Invalid argument"

-----------------------------------------------------------------------
--------

Disk Space Consideration

Each record in the audit trail takes about 50 to 150 bytes depending on
what
mode is used and whether the verbose mode flag is specified. This means
that
1MB of data could contain about 6800 entries.
-----------------------------------------------------------------------
--------

Understanding the Output

It is important to specify what information should be reviewed while

auditing.
Although you can configure auditing to record events of interest, there
may
still be too much data to be useful when viewed all at once. The
auditselect
command can be used with auditpr to sort through volumes of information
and
pull out only that which is needed for a specific report. It can be
used to
pull all data from a specific time period, for a specific user, or for a
specific event, or any combination of these three.

For example:

/usr/sbin/auditselect -f /audit/pick \
/audit/trail | /usr/sbin/auditpr -v

The /audit/pick file reads as follows:

command == rlogin && \

time >= 08:00:00 && time <= 17:00:00 && \
data >= 04/01/96 && date <= 04/12/96

This command reports the use of the rlogin command within the specific
time
interval (8AM-5PM between April 1 and April 12).

The compressed trail data from the binmode auditing is not in the same
format
at AIX version 3.2 as it is in 4.1 or later. There is a utility to
convert the
data from a pre-AIX Version 4 format to the Version 4 format. It is a
command
called auditconv.
-----------------------------------------------------------------------
--------

Common Problems with Auditing

Errors When Starting Audit

o There are certain errors that appear when running audit start.

Error Message:

** failed setting kernel audit objects

This occurs when there is a syntax error in the

/etc/security/audit/objects
 file.
o Error Message:

auditbin: ** failed backend command

/etc/auditcat -p -o /audit/trail -r /audit/bin1

This error can be corrected by removing or renaming the BIN files.

It is
sometimes helpful to run audit shutdown again and then to retry
audit
start.
o Error in config file:

It is necessary to have the "user" stanza in the

/etc/security/audit/
configuration file or the following error will display when you
start
auditing:

Unable to find the user stanza in /etc/security/audit/config

If it is not obvious that the user stanza is missing, verify that each
of the
classes are defined on a single continuous line.
-----------------------------------------------------------------------
--------

Data Overload

Given the way that cron and the TCPIP code is written, each sets up its
own set
of audit events. These events will get written into the audit trail
regardless
of how the config files are set up. The workaround is to use
auditselect to
exclude these events when generating the audit report. TCPIP sessions,
ftpd,
rexecd, and telnetd all call auditproc() to set up process auditing
using the
class tcpip in /etc/security/audit/config. The same thing is done in
the cron
code (at, cron, and cronadm) for the cron class in
/etc/security/audit/config.
These events will be written into the audit trail. The best thing to do
is to
filter them using auditselect.

For example:

auditselect -e"event!=AT_JobAdd && event!=AT_JobRemove && ..."

This will exclude events AT_JobAdd and At_JobRemove and so on.

Or select on the command name:

auditselect -e"command!=cron && command!=at && ..."

This excludes recording related to commands, at, cron, and so on.
-----------------------------------------------------------------------
--------

Fixes for Auditing Subsystem

As defects arise, updates to the AIX audit code are made.

Techdocs Ref:90605195614808 4FAX Ref:4824

Setup of System Accounting in AIX 3.2 [acct.setup.32.cmd]

Setup of System Accounting in AIX 3.2

-----------------------------------------------------------------------
--------

Contents

About This Document

Related Documentation
What to Install to Get Accounting Working on a System
Steps for Modifying the Root User Profile
Steps for Setting Up System Accounting
Fields of the Daily Usage Report
Fixes and Problems
Diagnosing Problems

-----------------------------------------------------------------------
--------

About This Document

This document describes the accounting setup procedures for AIX Version
3.2.

The setup of system accounting as documented in the 3.2 AIX

InfoExplorer has
some errors. This document describes the Version 3.2 setup procedures
and
provides the corrections to enable system accounting under AIX version
3.2.

Access to the InfoExplorer and system documentation for AIX Version 3.2
require
the setup person to be root. In this document, root user will be used
for
setting up system accounting and the user adm for running the reports.

Modify the root user .profile to provide access to the required

executables.
Modification of the root profile is optional, but will probably help
with
administration.
Related Documentation

InfoExplorer

-----------------------------------------------------------------------
--------

What to Install to Get Accounting Working on a System

Make sure the following product is installed:

bosext2.acct.obj - 'lslpp -l bosext2.acct.obj'

This command should report the state as COMMITTED or APPLIED if the

fileset is
installed. To get the latest level accounting software, refer to the
section
"Fixes and Problems" later in this document.
-----------------------------------------------------------------------
--------

Steps for Modifying the Root User Profile

1. Log in as root.
2. Edit the .profile file with your favorite text editor.
3. Locate the PATH statement in the .profile file. It must include the
/usr/sbin/acct and /var/adm/acct directories as shown in the
following
example:

PATH= /usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:
/usr/sbin/acct:/var/adm/acct ; export PATH

4. Locate the export statement in the .profile. If the PATH statement

given
above was not used, the PATH variable must be included.

-----------------------------------------------------------------------
--------

Steps for Setting Up System Accounting

1. Log in to the system as root user. As described previously, root

is the
accounting administrator.

2. Enter:

su - adm -c /usr/lib/acct/nulladm /var/adm/wtmp /var/adm/pacct

This command ensures correct permissions and provides access to

the wtmp
and pacct files.
3. Update the /etc/acct/holidays file for the current year using a
favorite
text editor. The following notes will help clarify the format of
this
 file:
o An asterisk (*) in column 1 denotes a comment.
o Define prime time on the first data line (the first line that
is not a
comment) using a 24-hour clock. The line will consist of three
4-digit
fields, in the following order:
- current year
- when (hhmm) prime time begins
- when (hhmm) prime time ends

Leading blanks are ignored. Midnight can be entered as either

0000 or
2400.

For example, to specify the year 1992 with prime time

beginning at
8:00 a.m. and ending at 5:00 p.m., specify:

1992 0800 1700

o Define the company holidays for the year on the next data
line. Each
line contains four fields in the following order:
- day of the year
- month
- day of the month
- description of the holiday
The day-of-the-year field contains the number of the day of
the year
in which the holiday falls. It must be a number from 1 through
365
(366 on a leap year like 1992).

4. The file systems that will be included for accounting information

need to
be configured in the /etc/filesystems file. For each file system
that will
be included, add the following information to its stanza:
account = true

5. Indicate the data file that will be used for printer data by
adding the
following line to the /etc/qconfig stanza for that printer:
acctfile = /var/adm/qacct

This entry must be added to the queue stanza and not the device
stanza. If
the entry is added by editing the /etc/qconfig file, enter the
following
command to re-digest the qconfig file:

enq -d

Another method of adding this entry to the qconfig file would be

to enter
the command smitty chque. Then follow the next steps:
 1. Select queue name.
2. Enter /var/adm/qacct for the ACCOUNTING FILE pathname.

These steps will update the appropriate stanza in /etc/qconfig

file and
will re-digest the qconfig file.

NOTE: The printer accounting will not record usage for network
printers,
transparent printers, or PostScript printers. In addition, all
print
queues must use the same accounting file:
/var/adm/qacct

The pac command (printer accounting command) is more versatile

because it
allows separate accounting files for each printer. For more
information,
see InfoExplorer.
6. Create directories /var/adm/acct/nite, /var/adm/acct/fiscal, and
/var/adm/acct/sum if they do not already exist. These directories
should
be owned by the user adm with a group ID of "adm". Use the
following
series of commands to create these directories:

su - adm
cd /var/adm/acct
mkdir nite fiscal sum
chown adm:adm nite fiscal sum
chmod 755 nite fiscal sum
exit

7. Modify the crontabs file for the adm user for automated
accumulation of
accounting data. The crontabs file for the adm user is in
/var/spool/cron/crontabs/adm.

To edit the adm crontab, log on as root and enter:

su - adm -c crontab -e

This will bring up the adm crontabfile in a vi session. You may

need to
uncomment or add some entries, and times may need to be modified
to suit
the installation's time requirements. An example of the entries
needed is
as follows:

10 23 * * 0-6 /usr/lib/acct/runacct 2> \

/usr/adm/acct/nite/accterr > /dev/null
0 23 * * 0-6 /usr/lib/acct/dodisk > /dev/null 2>&1
0 * * * * /usr/lib/acct/ckpacct > /dev/null 2>&1
15 4 1 * * /usr/lib/acct/monacct > /dev/null 2>&1

o The first line starts runacct at 11:10 pm (10 23), each Sunday
through
Saturday (0-6). runacct runs the daily reporting.
o The second line starts disk accounting at 11:00 pm (0 23),
each Sunday
through Saturday (0-6), before starting runacct.
o The third line checks the /var/adm/pacct file to ensure that
it does
not get too large at 0 minutes past each hour (0 *) every day
(*).

If the free space in /var falls below 500 blocks, then ckpacct
turns
off accounting until space is made available. A loss of
accounting
data will result during the period that accounting is turned
off.
ckpacct will turn accounting on again when more space is
available.
THERE IS NO NOTIFICATION unless the MAILCOM variable is set as
follows:

MAILCOM="mail root adm"

The variable can be set in the ckpacct and runacct scripts or

in the
/etc/environment file. If MAILCOM is set in both places the
setting in
ckpacct and runacct will be used.
o The fourth line runs the monthly accounting reports at 4:15 am
on the
first of every month.

8. Turn on process accounting by adding the following line to the

/etc/rc
file.
/usr/bin/su - root -c /usr/sbin/acct/startup

The startup command records the time that accounting was turned on
and
cleans up the previous day's accounting files.

If the system is not going to be rebooted at this time, run the

preceding
startup command from the root command line to start process
accounting.

-----------------------------------------------------------------------
--------

Fields of the Daily Usage Report

The meaning of the fields in the accounting Daily Usage Report are as
follows:

UID
User ID number
LOGIN NAME
 Login name of user
CPU PRIME
Cumulative CPU minutes during prime hours
CPU NPRIME
During non-prime hours
KCORE PRIME
Cumulative minutes spent in the kernel during prime hours
KCORE NPRIME
During non-prime hours
BLKIO PRIME
Cumulative blocks transferred during prime hours
BLKIO NPRIME
During non-prime hours
RW/WR PRIME
Cumulative blocks read/written during prime hours
RW/WR NPRIME
During non-prime hours
CONNECT PRIME
Cumulative connect time (minutes) during prime hours
CONNECT NPRIME
During non-prime hours
DISK BLOCKS
Cumulative disk usage
PRINT
Queuing system charges (pages)
FEES
Fee for special services
# OF PROCS
Count of processes
# OF SESS
Count of login sessions
# OF SAMPLES
Count of count of disk samples

-----------------------------------------------------------------------
--------

Fixes and Problems

Fixes for AIX Version 3.2.5 and later can be downloaded via the
Internet with
the FixDist service.

On very large systems, if there are over 1000 users in the /etc/passwd
file,
the /usr/sbin/acct/dodisk script must be changed. At line 136 in
dodisk, the
diskusg command must be changed to read as follows:

diskusg -Unnnnn $args > dtmp

nnnnn is the new maximum number of users for which disk accounting
should be
done. (When the maximum number is too small, the dodisk script will
return an
error message that tells the user to use -uxxxx. It should say -Uxxxx.)
On AIX 3.2.5 and Greater

o IX38748 corrects the problem of a user missing the output from

individual
users for the CPU and DASD output.
o IX39408 corrects the problem of process accounting failing on a
diskless
workstation.
o IX42292 corrects the problem of the runacct account command
truncating the
wtmp file during its nightly run, causing connection accounting to
be lost
for users who do not log off.
o IX40232 corrects the problem of daily and monthly per-user memory
accounting being off by a factor of exactly 200.
o IX42322 corrects the problem of a 64MB limit on system process
accounting.
With this problem, any process that used more than 64MB of memory
caused an
overflow, and the process size reported by the ac_mem field was
inaccurate.
o IX43161 corrects the problem of acctprc2 core dumping with more
than 500
users.
o IX41228 corrects the problem of acctdisk entering into an infinite
loop if
the input file is corrupted.
o IX40305 corrects the problem of the /var/adm/acct/sum/login log
file not
being updated in 3.2.5.

-----------------------------------------------------------------------
--------

Diagnosing Problems

General Information Needed

o Which accounting command is being used?

acctcms acctmerg
acctcom acctprc
acctcon acctprcl
acctdisk acctprc2

All of these accounting commands accept input from standard input

(as in
acctcms < /usr/adm/pacct) and redirect to standard output (as in
acctcmd <
/usr/adm/pacct > /tmp/report). Find out which accounting file is
being used
as standard input and where the output is being directed by
entering:

acct_cmd < In_file > Out_file

o Defaults are standard in and standard out. Exact syntax is very

important.

o How is accounting started, via cron or via command line?

If accounting is started via cron, note the crontab file entries. If

accounting is run from root, then the entry is in
/usr/spool/cron/crontabs/root. If accounting is run from adm, then
the
entry is in /usr/spool/cron/crontabs/adm.

When Accounting Fails

1. Check to see what state accounting is in. Look at the

/usr/adm/acct/nite/active file. States are as follows:

setup
wtmpfix
connect1
connect2
process
merge
fees
disk
queueacct
mergetacct
cms
userexit
cleanup

2. Check accterr for additional messages.

3. Refer to section "How To Fix General Accounting Problems" in
InfoExplorer.
4. Restart runaccount.

Other Items to Check

o Is the /usr directory out of space?

o Does wtmp have records with inconsistent data stamps?
o Is cron failing? (Check mail for root and adm.)

Techdocs Ref:90605194214786 4FAX Ref:2486