You are on page 1of 30

Technical White Paper

Centralised logging with rsyslog By
Peter Matulis – September 2009
© Copyright Canonical 2009

www.canonical.com
©

.

.

will be able to cover its traces by removing the portions of the logs that he wants. the attacker.canonical. having gained root access. Performing such analysis on each system is time consuming and is relatively insecure because if a server is compromised. Moreover. regulations and best practices often require the IT department to maintain an accurate log of all events happening in their systems in order to allow for later analysis. This white paper analyses the tools available in Ubuntu to perform the task of centralised logging.com . Centralised logging with rsyslog 2 www. recommends the use of rsyslog and provides the steps needed to configure a set of servers to send their events to a central logging facility.Overview The management of multiple systems requires the setup of tools to control the servers behaviour in real time and post analysis.

.

.

..........................................................9 Database logging................................. Multiple systems (to disk).......................................16 Queue processing..................................5 Logging models..................................8 Technical considerations to central logging.........20 On the logging server....................11 Installation...................................................................... Multiple systems (to database)................................................................................10 Getting started with rsyslog..................................................................................................................................................................23 Disk Queues............................................................................23 Centralised logging with rsyslog 3 www...........................................9 Logging software......................... Branch offices (remote storage)...............17 Central logging scenarios...................................6 1.................................................................................................................Table of Contents Overview......................................6 2..........................................9 TLS connections...................... Single system (to disk)....................................................................................14 Timestamps.......................................................canonical.................................................................................2 Introduction.........................................................................................................................................................14 Templates..................23 Logging queues...............................................................................................................................................................................................22 Advanced Rsyslog features applicable to central logging........................................................................................................21 On a logging client..............................................................................................................................18 Multiple systems (to database)..........................................................................................................................12 Output file syncing..............................................................................................................................................................................................................18 Multiple systems (to disk).............................18 Branch offices (remote storage)..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................com ......12 Rules/actions....................................................................................................................................................................................................................................................................14 Property-based filters.................................23 BSD-style blocks............20 On the Certificate Authority......................................6 3.9 Network logging reliability.11 Configuration structure.......................................................................................................................................................7 4.............................

.

.

.........................................25 Discard watermarks.......................................................................25 Remote database logging.........................................................................................30 Appendix D: Property options............canonical.......................................................................................................................In-Memory Queues........28 Appendix C: Message properties.............................................................................................................................27 Appendix B: rsyslog...............32 Centralised logging with rsyslog 4 www.........conf diff............................................................................................................................................25 Local disk logging................................................................................................24 Hybrid Disk-Assisted In-Memory Queues........................conf / syslog.........................................................com ......26 Appendix A: References and useful Links.............................................................24 Queueing and de-queueing.........................................................................................................................................25 Remote disk logging.......................................................24 Logging queue examples...........................

.

.

since logging is essentially a record of what is happening. In large organisations. who should have access to such logs?) ○ Legally. with systems processing company-sensitive or even secret data the subject assumes a whole new dimension.10 (karmic koala) is in alpha and uses rsyslog as its default tool for logging. regardless of its operating system. Geographically diverse branch offices bring another element to the mix. has a mechanism that records activities taking place within it. This paper is not an introduction to the field of system logging.canonical.Introduction Every computer system. Ubuntu 9.e. how far back in the past must a company retain its logs (particularly when managing client data)? The software that is covered in this document is rsyslog.com . "The Ins and Outs of System Logging Using Syslog" for the basics. This paper describes the reasons for the choice of rsyslog in the section Logging Software and provide technical caveats and background information in the section Technical Considerations and Historical Background. Centralised logging with rsyslog 5 www. Other questions deserving of serious consideration but which are not covered by this technical paper are: ○ Authorisation (i. where the number of computer systems can range in the thousands. Note: at the time of publication. In addition. replacing sysklogd that was the previous default . Possible alternatives are the stock Linux/Unix syslog system or syslog-ng. Finally. The analysis performed for this white paper is what triggered this change. logs play a vital role when a system has been compromised by an external (or internal) hostile agent. See Appendix A. Such 'logs' do not normally concern the average end-user but play a crucial role in the life of a system administrator or support analyst when problems arise because errors that occur are included in those activities and are the first stop in attempts at troubleshooting. This white paper also tries to address how a company technically manages the potentially huge volume of logs its computer systems generate. there is the task of managing such logging data.

.

.

com Single system _ "N 4* CD _.canonical. many systems forward their logs over the network to a central logging server. perform logging. Multiple systems (to disk) Known as central logging. Messages typically get written to the local hard drive but Network Attached Storage (NAS) or Storage Area Network (SAN) are also valid storage options for this model. on the server-side.Logging models This section surveys several typical architectural models of computer system logging. 1.J Logging server [ta disk] Multiple systems . 2. Analogous to the single-system model. Single system (to disk) Individual computer systems. Centralised logging with rsyslog 6 www. messages get written to the local hard drive or to some other available storage. by default.

.

.

a webbased interface acting as a viewing/query tool. Multiple systems (to database) A common option is to have the remote messages stored directly into a database on the server with. it can be placed onto a separate system.canonical. possibly. Centralised logging with rsyslog 7 www.3.com Multiple systems . The database need not reside on the logging server (as shown in the diagram).

.

.

relay EIFFICE B . Their central logging servers now relay their logs to a second-level central logging architecture (typically residing at the company head office or data centre). The fact that sensitive information is being transported over a non-trusted network (here the internet) is a vital facet that needs to be addressed by your company's security team.canonical.4.com INTERNET . Branch offices (remote storage) We continue the logical progression where multiple branch offices are each implementing the #2 or #3 model. Centralised logging with rsyslog 8 www.

.

.

Multiple servers may need to be used in such an environment. This is a great improvement but there remains nonetheless a reliability issue even with TCP. You therefore cannot at this time.com . The usage of RELP however is outside the scope of this white paper however. Database logging When logging to a database you need to make sure your database can actually handle the rate of messages (messages per second. use a dual-mode setup (encrypted and unencrypted channels). Centralised logging with rsyslog 9 www.canonical. Alternative software such as syslog-ng and rsyslog include support for the TCP protocol. Rsyslog offers ways to handle this and will be mentioned in the Advanced Rsyslog Features section but the database will often present an I/O bottleneck. Thousands of messages can be lost if the network connection with the logging server breaks as there is no mechanism in TCP that notifies the sender immediately (its send buffer continues to fill up). This is unsuitable for central/network logging due to the protocol's lossy/unreliable nature. The rsyslog project is currently developing a truly reliable logging protocol: RELP (Reliable Event Logging Protocol).Technical considerations to central logging Network logging reliability Traditional Unix syslog uses the UDP protocol. mps) that is being directed at it. TLS connections The TCP listener can currently only listen on a single port.

.

.

Compliance with IETF regarding reliable TCP transport (RFC 3195) Rsyslog is compliant with the standards regarding reliable TCP transport whereas syslog-ng is not. Native support for email alerts Rsyslog natively supports the ability to send email alerts based on log message content. Truly reliable message delivery (RELP) Rsyslog is confronting the unreliability of TCP in a logging environment through the development of the RELP protocol whereas syslog-ng is not. SNMP support Rsyslog supports SNMP traps whereas syslog-ng does not. It's unknown how these forks will diverge in the future. A commercial product has been forked from the open. Licensing and software features Syslog-ng is dual-licensed. 5. Centralised logging with rsyslog 10 www. 4.e. 2. 3. BSD-style hostname and program name blocks Rsyslog supports powerful BSD-style hostname and program name blocks for easy multi-host implementations whereas syslog-ng does not. 9. 6. This allows one to organize and split one's configuration into multiple files.source (GPL) project and the more advanced features are found only in the commercial offering. On-disk message spooling Rsyslog has on-disk file spooling features that are lacking in GPL syslog-ng: ● on-demand (as needed) spooling ● independent spool files per log action ● spool files on multiple disks ● Process spooled messages during configured timeframes 8.Logging software The rsyslog tool was chosen over the more popular syslog-ng for the following reasons: 1. not using stunnel) and ii) on-disk spooling of messages. Include config files Rsyslog has configuration include file support that syslog-ng lacks. Native support for traffic encryption (TLS/SSL) Rsyslog supports TLS natively whereas the GPL fork of syslog-ng does not. Affected features of import so far are i) native TLS/SSL support (i.canonical. 7.com . Syslog-ng needs to pipe data to an external process.