1

Log Blindspots: A review of cases where System Logs are insufficient

An ObserveIT Whitepaper | Brad Young

Executive Summary
If you spend a few minutes browsing the websites of Log Management and SIEM tool vendors, you might come away with the conclusion that all your system audit and compliance problems are solved. Unfortunately, this rosy picture seems to ignore the ever-present problem of blindspots in audit reports: If your apps dont log it, your audit report wont show it. Audit report tools may do a good job at interpreting and presenting log info, but we can no longer overlook two key facts: 1. Hundreds of critical security event types are not logged at all 2. Those events that are logged typically do not show what was done. Instead, the logs only show obscure technical details of the resulting system changes. In this whitepaper, Ill highlight examples of where these blindspots occur, by showing a number of very common and basic system activities that one might think should generate auditable log entries, but in actuality they do not. These non-audited actions include: On a Windows server: Adding and Deleting IP Address Setting a Service to run as administrator Change Web server config file Change port usage for an active service On a Linux or Unix server: chmod * or chown * Assign user to an admin rights group Add/Delete IP address in hosts file Give sudo rights to non-admin user

One possible way to eliminate blindspots is to implement custom log utilities or WMI-based tools. But to do this, the burden remains on you to know what you are looking for. For the examples listed above, adding an IP Address change monitor wont help with web config file changes, and vice versa. And more importantly, adding 4 different monitors for each of those issues wont help capture the hundreds of actions that youll never be able to predict. As the well-worn yet valuable expression states, Expect the unexpected. User Activity Monitoring follows through on this philosophy. In the context of IT audit logs, perhaps the best way to expect the unexpected is to drop the paradigm of listing the actions that should be logged, and instead simply monitor all user actions.

Log Blindspots: A review of cases where system logs are insufficient

Copyright 2011 ObserveIT Ltd. | www.observeit-sys.com

Scenario 1: Changing a Windows systems IP address

What the User Did: A privileged user logged onto a Win 2003 Server (via RDP in this case, but for the sake of discussion
it could be any local or remote connection protocol). After logging in, this admin user opened the Advanced TCP/IP Settings (via Start > Settings > Network Connections > Local Area Connection > Properties > Internet Protocol > Properties > Advanced). Once there, he removed an IP address (10.1.200.178), and then added a different IP address (10.1.200.179).

Advanced TCP/IP Settings

Security and Audit Implications of this Action: Adding and IP

address might allow bypassing of firewall settings and may also interfere with proper execution of critical services.

What shows up in system event logs: With full auditing enabled, a total over 11,000 log events were triggered
during the 30 seconds it took the user to delete and add an IP address. Almost all the log entries were of Object Access category. Searching within the logs for the terms TCP, IP or 179 (last 3 digits of the IP address added) brought back numerous search, but all were false hits. (ex: IP appears in the filename wshtcpip.dll within one log entry, another log entry having Operation ID 74312179.) No log entry refers explicitly to the action taken. It may be possible for a highly-trained system security expert to piece together the log entries and determine what actions took place. But it would involve a time-intensive forensic analysis by a sparse and expensive resource. Do you have highly-trained security experts that are bored with nothing better to do than piece together log entries?
Event Viewer: 11,000 log entries in 30 seconds, dozens of false hits, no clear picture

What User Activity Monitoring shows you: A

user-oriented textual audit log shows that brad logged on as administrator, and the list of actions tells the story of what he did: Network Connections > Properties > TCP/IP Properties > TCP/IP Address. This already is much more than information than what is accessible in the system logs. Adding video replay of the session then shows even more details.

ObserveIT Audit Log: A Table of Contents of the user session

ObserveIT video replay of user changing the IP Address

Log Blindspots: A review of cases where system logs are insufficient

Copyright 2011 ObserveIT Ltd. | www.observeit-sys.com

Scenario 2: Adding sudo rights for non-authorized users in Linux

What the User Did: A non-privileged user tried running the snmpd service, but did not have permissions. He then tried
running it using sudo, but did not have sudo rights either. So instead, he asked a root user to log on and grant him sudo rights, using visudo.

Add sudo rights for a non-authorized user

Security and Audit Implications of this Action:

Giving sudo rights allows a user to run many sensitive commands or services.

What shows up in system event logs: Using auditctl

and ausearch, we can see that the visudo command was run. However, this logging is almost entirely of a technical nature. We can see the working directory from which it was launched, its process id, and the fact that it finished with a success return value. No indication shows what rights were granted, or what the user did once he got those rights.
Technical details only in ausearch

What User Activity Monitoring shows you: With ObserveIT in place, we are able to see exactly what took place.
The textual metadata log shows the commands that were run.

ObserveIT Audit Log, including underlying system calls

ObserveIT video replay of CLI activities

Log Blindspots: A review of cases where system logs are insufficient

Copyright 2011 ObserveIT Ltd. | www.observeit-sys.com

Scenario 3: Setting a Windows service to run as administrator

What the User Did: An admin user changed the
properties of a Service (via Start > Settings > Control Panel > Administrative Tools > Services). Once there, he selected the Cryptographic Services service and marked it to run as administrator.
Run a service as Administrator

Security and Audit Implications of this Action: Enabling a service that is not secure to run as administrator can
enable remote hacking and can cause the service improperly affect sensitive system configuration and data.

What shows up in system event logs: Over 24,000 log events were triggered during the 40 seconds it took the user
to change the Run As credentials. Despite the sheer volume, no log entries included the word Cryptographic (the name of the service)! Again, a full-throttle investigation by system experts might unearth the true actions, but this task makes biblical archaeology look easy.
Event Viewer: 24,000 log entries in 40 seconds, no indication of the Service that was modified

What User Activity Monitoring shows you: As in the

previous example, ObserveIT shows a clear chronological timeline of what the user actually did: open Control Panel and then go to Cryptographic Services Properties. And again, video replay shows even more.

ObserveIT Audit Log

Video replay of Service Run As credentials

Log Blindspots: A review of cases where system logs are insufficient

Copyright 2011 ObserveIT Ltd. | www.observeit-sys.com

Scenario 4: Change web.config (IIS webserver configuration file)

What the User Did: Via Windows Explorer
and Notepad, the user made a simple change to an XML attribute in the file web.config, changing a 0 (false) value to 1 (true).

Editing web.config with Notepad

Security and Audit Implications of this Action: Changes to this file will affect
how the web server runs, in numerous different ways. This can expose security risks, and can also affect proper operations.

What shows up in system event logs: 6,000 log entries cover the 20 seconds it took to make the change. One log
entry indicates that Notepad was launched. Another log entry indicates that web.config was added to the Recent Files list in Windows. A third log entry seems to show (not convincingly) that it was Notepad that edited the filw web.config. But even with this info, we cannot tell what was actually changed within the file! (Was it a harmless addition of an application extension? Or did the user modify an important entry within the file?) To know what was changed, we would now have to access a file server backup, and perform a file compare on the old and new versions. Doable, but thats a heavy burden to answer a pretty straightforward question: What did the user change???
Event Viewer: But what was changed?

What User Activity Monitoring shows you: ObserveITs log shows what the user did, in a concise and descriptive
manner. And again, video replay shows what took place within the file.

Log Blindspots: A review of cases where system logs are insufficient

Copyright 2011 ObserveIT Ltd. | www.observeit-sys.com

Scenario 5: Changing the port used by IIS

What the User Did: An admin user changed IIS to
listen to port 8080 instead of the default 80. This was done via Start > Settings > Control Panel > Administrative Tools > IIS Manager, and once there editing the Properties for Default web site.
Set IIS to listen to port 8080

Security and Audit Implications of this Action:

Modifying the port of a service accessible from outside the DMZ can open a huge hole in the firewall security.

What shows up in system event logs: Among the 5,500 log entries, there is one entry that adds IIS Manager to the
Recent Items list in Windows. This is timestamped when the app was closed, which might mislead the investigator, and alsow wouldnt even occur if the user left the window open. Earlier, there is an obscure log entry indicating a DLL that was loaded to memory. This is the true indication that IIS Manager was launched, but it is very difficult to find this in a reasonable level of effort!
Event Viewer: Obscure log entry of DLL. It turns out that this is the culprit!

What User Activity Monitoring shows you: Once again, ObserveIT gives us the whole picture.

Log Blindspots: A review of cases where system logs are insufficient

Copyright 2011 ObserveIT Ltd. | www.observeit-sys.com

Platform Considerations
The Windows experiments were performed on a Windows 2003 server. Windows 2008R2 has added additional audit policy granularity. However, these updates do not mean that additional knowledge can be gleaned from the logs; Only that the logs can be filtered a bit better. The bottom line remains that many high-risk, security-impacting actions, including those highlighted in this paper, are not logged. The Linux experiments were performed on RedHat RHEL. Similar audit logging is found in other Linux flavors, as well as in Solaris Unix, with similar focus on technical aspects of each command (pid, cwd, success).

Conclusion
Security audits that rely on existing system logs have large holes in them due to the fact that system logs simply do not capture the relevant information necessary. For issues that are known a priori, the blindspot can be eliminated with a custom utility targeted at that specific issue. But this only solves this one specific issue. The easiest way to eliminate these blindspots in their entirety is by adding User Activity Monitoring such as ObserveIT, which augments the existing system and database logs by showing precisely what the user did (as opposed to the technical results of what he did.)

About ObserveIT
ObserveIT User Activity Monitoring software meets the complex compliance and security challenges related to user activity auditing, one of the key issues that IT, Security and Compliance officers are facing today. ObserveIT acts like a security camera on your servers, generating audit logs and video recording of every action the user performs. ObserveIT captures all activity, even for applications that do not produce their own internal logs. Every action performed by remote vendors, developers, sysadmins and business users is tied to a video recording, providing bulletproof forensic evidence. ObserveIT is the ideal solution for 3rd Party Vendor Monitoring, and PCI/HIPAA/SOX/ISO Compliance Accountability. Founded in 2006, ObserveIT has a worldwide customer base of Global 2000 companies that spans many industry segments including finance, healthcare, manufacturing, telecom, government and IT services.

For more information, please contact ObserveIT at: www.observeit-sys.com sales@observeit-sys.com US Phone: 1-800-687-0137 Intl Phone: +972-3-648-0614

Log Blindspots: A review of cases where system logs are insufficient

Copyright 2011 ObserveIT Ltd. | www.observeit-sys.com