Host Intrusion Detection System (HIDS) v3.

1 Sizing Guidelines and Tuning Primer September 2005

HP Company Internal

Page 1 of 20

Legal Notices
The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. Warranty. A copy of the specific warranty terms applicable to your Hewlett-Packard product and replacement parts can be obtained from your local Sales and Service Office. Restricted Rights Legend. Use, duplication or disclosure by the U.S. Government is subject to restrictions as set forth in subparagraph (c) (1) (ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013 for DOD agencies, and subparagraphs (c) (1) and (c) (2) of the Commercial Computer Software Restricted Rights clause at FAR 52.227-19 for other agencies. Hewlett-Packard Company 19420 Homestead Road Cupertino, California 95014 U.S.A. Use of this manual is restricted to this product only. Copyright Notices copyright 1983-2005 Hewlett-Packard Development Company, LP. Reproduction, adaptation, or translation of this document without prior written permission is prohibited, except as allowed under the copyright laws.

HP Company Internal

Page 2 of 20

Contents

1.1 PRODUCT IDENTIFICATION ............................................................................................... 4 1.2 PURPOSE OF DOCUMENT .................................................................................................. 4 1.2 INTENDED AUDIENCE ....................................................................................................... 4 1.3 GLOSSARY ....................................................................................................................... 4 2.1 PRODUCT OVERVIEW ....................................................................................................... 5 2.2 HP-UX HIDS DEPLOYMENTS ......................................................................................... 5 2.3 SIZING AND TUNING OVERVIEW ...................................................................................... 5

2.0 OVERVIEW...................................................................................................................... 5

3.0 SIZING AND TUNING RECOMMENDATIONS ........................................................ 6 3.1 SIZING GUIDELINES ......................................................................................................... 6 3.1.1 Single vs. Multi-Processor ....................................................................................... 6 3.1.2 Number of CPUs ...................................................................................................... 6 3.1.3 Memory .................................................................................................................... 6 3.1.4 Disk Capacity........................................................................................................... 7 3.2 TUNING CONSIDERATIONS ............................................................................................... 7 3.2.1 Product Tuning ........................................................................................................ 7 3.2.1.1 Tuning the Surveillance Schedules................................................................... 7 3.2.1.1.1 Background ................................................................................................ 7 3.2.1.1.2 Avoid duplicate copies of a template......................................................... 7 3.2.1.1.3 Avoid duplicate groups with overlapping functionality ............................ 7 3.2.1.1.4 Race Condition Template .......................................................................... 8 3.2.1.2 Tuning Process Priority..................................................................................... 8 3.2.1.3 Tuning the HIDS System Manager (GUI) ........................................................ 8 3.2.2 Kernel Tuning .......................................................................................................... 8 3.2.2.1 Tuning the Kernel Audit System (IDDS) ......................................................... 8 3.2.2.1.1 System performance over security............................................................. 9 3.2.2.1.2 Security over system performance............................................................. 9 3.2.2.1.3 How to change from non-blocking to blocking mode ............................... 9 3.2.2.2 Kernel Tunables ................................................................................................ 9 3.2.2.2.1 enable_idds ......................................................................................... 9 3.2.2.2.2 max_thread_proc................................................................................ 9 3.2.2.2.3 tcp_conn_request_max.................................................................... 9 3.2.2.2.4 secure_sid_scripts ........................................................................ 9 3.2.2.2.5 executable_stack ........................................................................... 10 3.2.2.2.6 maxdsiz................................................................................................. 10 3.2.2.3 Swap................................................................................................................ 10 4.0 REFERENCE DOCUMENTS/ WEB SITES ............................................................... 11 APPENDIX A CPU CONSUMPTION ............................................................................ 12 CPU Consumption on PA Processors ........................................................................... 13 CPU Consumption on Itanium Processors ................................................................... 15 APPENDIX B RESIDENT MEMORY CONSUMPTION ............................................ 17 Memory Consumption on PA Processors ..................................................................... 17 Memory Consumption on Itanium Processors ............................................................. 19
HP Company Internal Page 3 of 20

1.0 INTRODUCTION 1.1 Product Identification

Product Name: HP-UX HIDS Product Number: HPUX-HIDS Product Version/Release: 3.1

1.2 Purpose of Document

This document provides basic sizing and tuning guidelines for HP-UX Host Intrusion Detection System (HIDS). The sizing guidelines are generated using a purely artificial load-generating environment that generates a constant stream of system call audit records that HIDS processes (see Appendix A for details). Testing for these guidelines was performed on dedicated HP-UX servers. No other system activity was occurring during the tests. However, when deploying HIDS into production environments, be careful to assess system load generated by other applications, and factor the HIDS throughput requirements accordingly.

1.2 Intended Audience

The data provided in this document is intended to help customers effectively size and tune their systems running HIDS and to help the HP field force effectively size and tune customer configurations for deployment of HIDS.

1.3 Glossary
The following are definitions and acronyms used within this document. Definitions Agent - The HIDS sensor that detects intrusions. Event - Any piece of information that is being analyzed by HIDS for intrusions. For example, system call audit records and login records are all delivered to HIDS as events. Surveillance Group A collection of one or more template instances where each instance is of a unique template type. Surveillance Schedule A collection of one or more surveillance groups where each group has its own set of template instances. Template or Circuit Intrusion detection logic that analyzes events. Detects the use of basic attack building blocks or patterns. Template Instance An instance of a template. For example, there can be several instances of the Modification of Files/Directories template, each of which monitors for the modification of different critical files or directories. Template Type Specifies which template logic a template instance implements (e.g., Modification of Files/Directories). Template Properties Configuration {name,value} tuples that are used to parameterize a template instance and change a template instances behavior at run time. Two template instances of the same template type have the same property names but with potentially different property values. If properties are modified for a surveillance schedule that is running, the schedule must be restarted for the new property values to take effect. Acronyms CPU HIDS HP-UX IDDS Central Processing Unit Host Intrusion Detection System Refers to the HP-UX Host IDS product. HPs flavor of Unix Intrusion Detection Data Source - A kernel auditing subsystem on 11.11 and 11.23 specifically designed to provide a source of rich, on-line kernel audit data for HIDS.

HP Company Internal

Page 4 of 20

2.0 OVERVIEW
2.1 Product Overview
HP-UX HIDS is an HP-UX host intrusion detection product that can enhance local host-level security within your network. It does this by automatically monitoring each configured host system within the network for possible signs of unwanted and potentially damaging intrusions. If successful, such intrusions could lead to the loss of availability of key systems or could compromise system integrity. As HIDS continuously examines ongoing activity on a system, it seeks out patterns that might suggest security breaches or misuses. These might include, for example, an attacker attempting to break into or disrupt your system, subversive insider activities, or someone trying to spread a virus. Once you have activated HIDS for a given host system and it detects an intrusion attempt, the host sends an alert to the administrative interface where you can immediately investigate the situation, and when necessary, take action against the intrusion. HIDS also supports customized local responses to, for example, notify the administrator through e-mail or pager.

2.2 HP-UX HIDS Deployments

HIDS can be deployed on any HP-UX 11iv1 or 11iv2 server that contains applications and data that need to be monitored for protection and/or availability, such as web servers, transaction processors, application servers, and database systems. The performance of HIDS depends on the system load, the rate at which certain system calls are invoked by other applications, and the HIDS configuration.

2.3 Sizing and Tuning Overview

The following guidelines should be used when selecting a system to run HIDS. They are discussed in more detail in Section 3.0 Sizing and Tuning Recommendations. Templates, the component of HIDS that detects intrusions, are designed to take advantage of multiple CPUs, if available. The amount of memory and disk space needed depends on the system load profile and the HIDS configuration. Sustained high loads can consume large amounts of memory. When heavily loaded, CPU is the eventual performance bottleneck.

HIDS performance tuning is limited to: Surveillance schedule configuration Process priority setting

System performance tuning is limited to: Blocking vs Non-blocking IDDS mode

HP Company Internal

Page 5 of 20

3.0 Sizing and Tuning Recommendations

3.1 Sizing Guidelines
Any HP-UX platform that supports HP-UX 11iv1 or 11iv2 can be utilized to run HIDS. selecting a server platform for HIDS deployments, consider the following system parameters: Single vs Multi-Processor Number of CPUs Memory Disk Capacity When

Note: These sizing guidelines apply to servers running the HIDS agent sensor and not the HIDS System Manager (GUI).

3.1.1 Single vs. Multi-Processor

The component of HIDS that executes the intrusion detection logic is multi-threaded and therefore benefits from multiple processors. The benefit on multiple processor systems of allowing intrusion detection templates to run concurrently and therefore process events faster must be tempered with the following: More processors allows more applications to produce event loads that need to be consumed by the HIDS agent. The impact of the HIDS agent depends on the system call activity of the applications producing the load and therefore is highly server load specific. The benefit of more processors diminishes when the number of processors exceeds the total number of HIDS agent threads that process event loads. The total number of these HIDS threads is (T + 2), where T is the number of detection templates running and has a maximum value of 10 if HIDS is running only one instance of each template type.

3.1.2 Number of CPUs

For the majority of deployments, the performance bottleneck for HIDS will typically occur at CPU, primarily from the idscor process. The idscor process is multi-threaded and can therefore utilize over 100% CPU. HIDS will generally reach the CPU limit before other constraints such as disk or memory are realized. The CPU consumption by the HIDS processes is charted against the rate of system call audit records (events) in Appendix A.

3.1.3 Memory
As the sustained event load on the server is increased, a greater amount of resident memory may be consumed, especially by the idscor process that dynamically allocates heap memory to store and process events. On systems with a low amount of memory, or with memory contention with other applications, virtual memory/disk I/O (i.e., process swapping) can affect the performance in these circumstances. An additional 40 to 60 MB of memory is recommended for all of the HIDS agents processes.

HP Company Internal

Page 6 of 20

The memory consumption of the HIDS agent processes is charted against the rate of system call audit records (events) in Appendix B.

3.1.4 Disk Capacity

One of the main functions of HIDS is to log alerts locally to disk on the server being monitored. By default, the log file used is /var/opt/ids/alert.log. The amount of alerts will vary depending on what HIDS is configured to monitor and the load activity on the system. The continuous operation of HIDS can produce many alerts and can therefore consume a large amount of disk space. In addition, a 20 megabyte memory mapped file is created in /var/opt/ids. It is recommended to allocate at least 100MB to the disk partition that contains /var/opt/ids on each system running the HIDS agent. The amount of disk space needed can be mitigated by performing log rotation of alert.log. For swap, the HIDS agent requires between 97 MB and 157 MB.

3.2 Tuning Considerations 3.2.1 Product Tuning

3.2.1.1 Tuning the Surveillance Schedules

3.2.1.1.1 Background
A surveillance schedule contains one or more surveillance groups. A surveillance group defines a collection of detection templates, their corresponding configurations, and when the templates are scheduled to run. A detection template may exist in more than one surveillance group, but each surveillance group may have at most one template instance of a particular template. One can configure each detection template in the group with details specific to the threats to protect against. For example, a surveillance group named "WebServer" may contain three templates: Creating SetUID files, Changes to files/directories and Monitor logins/logouts. In this example, the Changes to files/directories template can be configured to monitor the changes to files and directories under /etc/opt/httpd.

3.2.1.1.2 Avoid duplicate copies of a template

It is possible to place the same detection template in two or more surveillance groups. However, if the groups are scheduled to run concurrently in a surveillance schedule then multiple copies of a detection template will be executing concurrently. A performance penalty will be incurred from running more than one instance of the same template. Try to schedule surveillance groups with duplicate templates to run at different times.

3.2.1.1.3 Avoid duplicate groups with overlapping functionality

A surveillance group should contain the least number of templates required to be effective, and no more. One can reduce the likelihood of duplicate templates by keeping surveillance groups as small as possible.

HP Company Internal

Page 7 of 20

3.2.1.1.4 Race Condition Template

The race condition template imposes the highest CPU and memory overhead on the system. Use this template with care if concerned about CPU utilization.

3.2.1.2 Tuning Process Priority The HIDS idscor process performs the CPU and memory intensive operation of executing the detection templates that process the events. For potentially better event processing throughput, one can allow the idscor process to run with a higher system priority by adjusting the process nice value (see the nice(2) man page).

3.2.1.3 Tuning the HIDS System Manager (GUI) Because the HIDS System Manager is a Java application that can be memory and CPU intensive, it is recommended to run the HIDS System Manager on a dedicated management server. The JVM can, for example, consume over 50% CPU when a user performs operations as simple as opening a new window. Under certain loads, such as when receiving thousands of alerts, the Java Virtual Machine (JVM) running the GUI might log out-of-memory errors in /var/opt/ids/gui/guiError.log or /var/opt/ids/gui/logs/Trace.log. The fix is to increase the maximum heap size of JVM (default is 64MB) in the /opt/ids/bin/idsgui file by adding, for example, -Xmx256m \ after the line "$JAVA_RUN \". In this example, the max heap size is set to 256MB.

3.2.2 Kernel Tuning

3.2.2.1 Tuning the Kernel Audit System (IDDS) HIDS monitors kernel audit data, specifically system call audit records. The rate of audit data generated on a system can vary dramatically depending on the system load. On a lightly loaded system that is idle, there may be a dozen system calls a second. However on a heavily loaded system that rate can increase to thousands of system calls per second. As the number of processors on the system increases, the rate of audit data generation also increases. The HIDS agent processes operate in user space and must read data from the kernel audit data source called IDDS. The kernel stores audit data records in a buffer until the agent process is ready to read them. If the HIDS agent process is not yet ready to read and the kernel buffer has no space left for an audit record, the kernel must make a choice: discard the data or block the process which generated the audit data until space becomes available in the buffer. The "non-blocking mode" defines the situation where the kernel discards the data before HIDS has a chance to read it. The name "non-blocking" refers to the fact that a process executing a system call will never block if no space is available in the kernel audit buffer. The "blocking mode" defines the situation where the kernel will suspend the process executing a system call until space becomes available in the kernel audit buffer. In blocking mode, a system call will not return until space is available in the buffer for the audit record.

HP Company Internal

Page 8 of 20

3.2.2.1.1 System performance over security

The default setting for an HIDS agent is non-blocking mode because, in certain cases, it is possible that blocking mode may have an overall negative impact on system performance. For example, one may find that many processes are suspended because the audit record buffer is full. The total system throughput may therefore be reduced. Use non-blocking mode if system performance takes precedence over security.

3.2.2.1.2 Security over system performance

In the blocking mode, no data is discarded before the agent can process it. As no data is discarded, there is less likelihood that an intrusion will be missed. Thus this setting places a premium on security.

3.2.2.1.3 How to change from non-blocking to blocking mode

The mode setting is controlled by the IDDS_MODE entry in the ids.cf configuration file (default location is /etc/opt/ids/ids.cf). The IDDS_MODE entry in the ids.cf file can be set to one of the following values: 2 - blocking mode 3 - non-blocking mode (default) The ids.cf file must be reread and any running HIDS surveillance schedule must be restarted before the change to ids.cf takes effect (no reboot is required). See the HIDS Administrators Guide in Appendix E for more details on configuring and rereading the ids.cf configuration file. 3.2.2.2 Kernel Tunables

3.2.2.2.1 enable_idds
This tunable is automatically set to 1 when HIDS is installed. This tunable must be set to 1 in order for IDDS to produce system call audit records that are needed by the HP-UX HIDS file related templates to detect intrusions. This tunable can be set to 0 to disable IDDS.

3.2.2.2.2 max_thread_proc
You need to ensure that the system on which the HIDS System Manager is running provides enough threads per process to handle the maximum number of agent systems you will monitor at one time. See Enabling Over 23 Agents (Thread Limits) in the Configuration Chapter of the HP-UX HIDS Administrators Guide for details.

3.2.2.2.3 tcp_conn_request_max
The HIDS System Manager communicates with agent systems using the TCP protocol. On some systems, the TCP parameter, tcp_conn_request_max, is set initially to allow up to 20 inbound requests to be active at one time. If you have a larger number of agent systems, this value will be inadequate. If this is a problem, an agents error log will contain messages like write_msg: error opening connection to remote host..., open_connection: connect error, and open_connection: Timed out waiting on select() for connect to complete. You can view and change this parameter with the ndd command. See Enabling Over 20 Inbound Requests in the Configuration Chapter of the HP-UX HIDS Administrators Guide for details.

3.2.2.2.4 secure_sid_scripts
Starting with 11i v1.6, the execution of setuid scripts, which is vulnerable to race condition attacks, is prevented if this tunable is set (enabled by default). Enabling this tunable will prevent setuid script

HP Company Internal

Page 9 of 20

race condition attacks, while the HP-UX HIDS Race Condition template will detect them. See the secure_sid_scripts(5) man page for details. Even if the secure_sid_scripts tunable is enabled to prevent setuid script attacks, you might still want to run the Race Condition template to detect other types of race condition attacks (see the Administration Guide in Appendix A for more details on what the Race Condition template detects).

3.2.2.2.5 executable_stack
Starting with 11i v1, this tunable provides comprehensive stack buffer overflow protection by using a combination of highly efficient software and existing memory management hardware. Enabling this tunable will prevent certain stack buffer overflow attacks, while the HP-UX HIDS Buffer Overflow template will attempt to detect them. See the executable_stack(5)man page for details.

3.2.2.2.6 maxdsiz
If the HP-UX HIDS agent error file (/var/opt/ids/error.log) contains out of memory errors, the maximum data segment size may need to be increased. 3.2.2.3 Swap If the HP-UX HIDS agent error file (/var/opt/ids/error.log) contains out of memory errors, the swap space may need to be increased. Run the /usr/bin/swapinfo command to determine your swap usage.

HP Company Internal

Page 10 of 20

4.0 Reference Documents/ Web sites

Refer to the Administrators Guide and Release Notes for the latest release at http://docs.hp.com. HP-UX HIDS can be downloaded from http://sofware.hp.com.

HP Company Internal

Page 11 of 20

Appendix A CPU Consumption

The charts below show the CPU consumption of all HIDS processes for various systems when an artificially created rate of system call audit records (events) are applied on the system and when certain HIDS templates are running. The File Templates include the Modification of Files/Directories, Creation of world-writable files, Creation and modification of SETUID files, Modification of another users files, and Changes to log files. The RC Template is the Race Condition template. To measure the average system call audit event rate on a system, you must run the idscor process with the -t option while running any of the file related templates. The idscor t option is not supported by HIDS v3.1 and a special v3.1 version of idscor along with documentation must be obtained through technical support. The t option will be documented and available starting with HIDS v4.0.

HP Company Internal

Page 12 of 20

CPU Consumption on PA Processors

The graphs below show that the CPU consumption of HIDS processes increases as the event load is increased.

2 Way PA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 50 CPU % 100 150

All Templates File Templates RC Template

4 Way PA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 20 40 60 CPU % 80 100

All Templates File Templates RC Template

HP Company Internal

Page 13 of 20

8 Way PA
30000 25000

Event/sec

20000 15000 10000 5000 0 0 50 100 150 200 250 CPU %

All Templates File Templates RC Template

16 Way PA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 50 CPU % 100 150

All Templates File Templates RC Template

HP Company Internal

Page 14 of 20

CPU Consumption on Itanium Processors

The graphs below show that the CPU consumption of HIDS processes increases as the event load is increased. For any given event rate, the CPU consumption is less than on PA systems by a significant amount (between approximately 30-80%).

2 Way IA
30000 25000
Events/sec

20000 15000 10000 5000 0 0 20 40 CPU % 60 80 100

All Templates File Templates RC Template

4 Way IA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 20 40 CPU % 60 80

All Templates File Templates RC Template

HP Company Internal

Page 15 of 20

8 Way IA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 20 40 CPU % 60 80

All Templates File Templates RC Template

16 Way IA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 20 40 60 CPU % 80 100

All Templates File Templates RC Template

HP Company Internal

Page 16 of 20

Appendix B Resident Memory Consumption

The charts below show the resident memory consumption of all HIDS processes for various systems when an artificially created rate of system call audit records (events) are applied on the system and when certain HIDS templates are running. See Appendix A above for the definition of which templates constitute the File Templates and how to measure the event rates on your system.

Memory Consumption on PA Processors

The graphs below show that the memory consumption of HIDS processes stays within 3-4 megabytes as the event load is increased, and in some cases, the memory consumption decreases as the event load is increased.

2 Way PA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 10000 20000 30000 40000 50000 60000 Resident Memory (KB)

All Templates File Templates RC Template

4 Way PA
30000 25000
Events/sec

20000 15000 10000 5000 0 41000 42000 43000 44000 45000 46000 47000 Resident Memory (KB)

All Templates File Templates RC Template

HP Company Internal

Page 17 of 20

8 Way PA
30000 25000

Events/sec

20000 15000 10000 5000 0 40000

All Templates File Templates RC Template

42000

44000

46000

48000

Resident Memory (KB)

16 Way PA
30000 25000

Events/sec

20000 15000 10000 5000 0 42000 43000 44000 45000 46000 47000 48000 Resident Memory (KB)

All Templates File Templates RC Template

HP Company Internal

Page 18 of 20

Memory Consumption on Itanium Processors

The graphs below show that the memory consumption of HIDS processes stays within 3-4 megabytes as the event load is increased.

2 Way IA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 20000 40000 60000 80000 Resident Memory (KB)

All Templates File Templates RC Template

4 Way IA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 20000 40000 60000 80000 REsident memory (KB)

All Templates File Templates RC Template

HP Company Internal

Page 19 of 20

8 Way IA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 20000 40000 60000 80000 Resident Memory (KB)

All Templates File Templates RC Template

16 Way IA
30000 25000

Events/sec

20000 15000 10000 5000 0 0 20000 40000 60000 80000 Resident Memory (KB)

All Templates File Templates RC Template

HP Company Internal

Page 20 of 20