swwwwwwVwwewwwwwwwYwwYUvvweuwwwwwsd Content ‘system Logging Reysiog Jeicirsysiog cont Log Management Log Anomaly Detector Lab Tasks 1, Setting up Full Debug Logfile 42, Remote Syslog Contiguration Chapter 14 LOG FILE ADMINISTRATIONImplementations ‘The otiginal Unix system log daemon was writen for Sendmail and ‘was 8 stendard part of Berkeley Unix (BSO). An enhancement called Sysklogd has been dominant on Linux systems untl recent years ‘This provided systogd and klogd, and was configured in Jete/systog.conf. It has been replaced on some systems with Sysiog-NG. However, all enterprise distributions now ship Reyslog, 8 drop in replacernent for Sysklogd. ‘Syslog Facilities and Priorities ‘The facity is used to specify what type of program is generating the ‘massage. The Syslog daemon can then be configured to hancle ‘messages from differant sources differently. Ths table lists the Standard fined acts with Bet deserpos of what the re Used for 142 ‘System Logging ‘System Messages "Generated by Apptenons or taal oon mann xing termite ylcy ard pasty * Logs system messages * Supports remote logging Kernel Log Daemon ‘= Receves messages from tho Linux kornal and sends them to the system log daemon (Fecitey [Description lauthlauthpriv | securty/euthorzation messages [exon [erond and atd daemons messages [daemon ether system daemons ker kernel messages [locale — local? [reserved for local use hier ing printer subsystems frat all subsystem ews USENET news subsystem aystog messages gonerated interaly by the system log daemon [user [generic userievel messages favep [UCP subsystem AAR RR RRR RRR RRR RRNA NN RNvwwwwwewwrwwwwwurvowwwwwwowowwwwved ‘The priority, of level, of a message is intended to determine the importance of a message. Ths table sts the standard priority levels with beet descriptions oftheir meanings os een oo Some Sc [ona area sa [eer Enabling Remote Logging The Syslog daemon isnot configured to listen on the network by default. On Red Hat Enterprise Linux systems make the following edit to enable remote logging: [Fis: fete/rystog.cont ¥ Provides UDP syslog reception ———————SSSCSCSC~*Y - | sstodbead-ieadp-s0 «| s6uoeGoeveraun-Si4 +|SHoatoad imudp.so +|Suppservernun $14 asReyslog Rayslag is a logging daemon intended to compete with Syslog-ng ‘When Rainer Gerhards, the pimary author of Rsysiog, began development of his logging daemon, Syslog-ng was already well-established as a logging daemon boasting many features lacking {rom the ubiquitous Syskiogd. When describing his motivation for Creating yet another altemative to Sysklogd, he explained his goal to Lge his creation to “prevent monocultures and provide a rich freedom ‘of choice." Raysiog has since become the default logging daemon for ‘most Linux distributions In order to ease the tansition from Syskiogd {0 Rsysiog, the existing conficurations wil continue to work wh using Rysiog. after which anv Rsyslog specific enhancements may be added. Reliable Network Logging Rather than being limited to LDP, like most logging daemons, Reyslog supports TCP over [Pv or IPV6. TCP is more reliable than UDP, Because of its acknowledgment and retransmission capabilities. ‘Another way Ryslog ean increase reliably is through the use of fallbeck logging destinations, When rsysloga is unable to log to @ Particular destination, one or more additional destinations, either hosts or fos, may be specified for message delivery 144 Reyslog Drop-in replacement for Sysklogd Ihx6 support Reliable fogaing ‘Logging over IP 1 Serer faover Precee logging Enhanced fitering 1 Procio umestamps Definon of moseage output formats ‘Ability to log to SOL databaree TLS cheryption| Precise Logging Reyslog also provides several enhancements over traditional logging ‘implementations with regards to logging pression, For exampl possible to fiter messages on any part of a log message rather the priority of the message and the originating faiity. Among other things. this allows for per-host logfile creation on the logging server ‘Additionaly ifs possible (using templates) te choose the exact logging format of messages created by Aisyslog, including support for ‘more precise imestamps then are used with standard Sysiog log messages. ‘Additional Reyslog Features syslog provides many additional features which are unavailable with ‘Syskiogd, such as support for TLS encryption, and the ability to log tO SOL databases. Fora full list of available fesures, as wal as a side-by-side feature comparison of Rsyslog and Syslog-ng, visit log con/ doom ‘i RRA RRR RRR RRR RR RAR RAR AA RRA AAR AAASwe VEU VEU VVUwwUUvUewweUTuuwovevWsd reyslog.cont {All system and kernel messages get passed to rsystogd. For every log message received Reyslog looks a its configuration Jete/syslag.cont to determine how to handle that message. Fysiog looks through the configuration file fo all rule statements ‘which match that message, and handles the message as each rule statement dictates. I no rule statement matches the message, Reyslog discards I. Rule statements specify two things: what ‘messages to match (selectors), and what to do with matched ‘messoges (actions). Selectors ‘Messages to match are specified by 3 selector which matches facilities and prorties, while actions to apply to matched messages ‘are specified by an action field. For example, the following Configuration fine tolls Rsysiog to apply the action /var/Log/kerntog toll messages witha facility of Kera and a level of debug: Fis Tete/ayatog. cont [Therm debug TvasTog/Raratog | Priority statements in selectors are hierarchical. Rsyslog will match all ‘messages of the specified priority and higher. The s ern. debug matches all messages produced by the priority dabug of higher; since debug is the lowest possible priority, the selector kern. debug matches ail messages with a kern facility. In dition, an asterisk can be used as 8 wild card to represent all letc/rsysiog.cont "Rules consist of slactor fields and aeton fed separated by ‘spaces of tabs ‘Mulplo selectors per line, separated by semicolons ‘Only one action per ne Selector 1 The selector consists ofa facity an a prot, separated by 8 porod, such a5 mail. info “An asta stands Tor al feces or al prices Aetlon ' Can log t: fle including devices), FIFO, terminal, ‘machine, specie users, all usrs priorities, so ker kernel + would also match all messages produced by the Unlike the priory field, the facility field isnot hierarchical. I is sti possible to match multiple messages from diffrent facities, however. Multiple selectors can be listed on a ine, separated by ‘semicolons. This can be useful when the same action needs to be ‘applied to multiple messages, Sinaly, the asterisk wild-card can be used to specify all facities, providing another method for applying an ‘action toa variety of messages. Actions Many actions are possible, though only one can be included in rule ‘File names can be listed in the action field, specifying the location of files to which the selected message should be written. These files can be text files, as is usually the case, but they can also be device files such asa terminal ora printer. 4 Usor names can also be specified. Il the named user Is logged ino the system when Rsysiog processes the message, the ‘message will be printed to al ofthat user's terminal. %An asterisk forthe action tolls Rsysiog to write the message to all logged:in users (it goes to al active terminals) 1 Messages can be sent to remate hosts. The action @host tells Reyslog to forward the message to the machine host, where it will be processed again by that host's Syslog daemon, 145Log Management {All system administrators must watch ther system logs on a regular basis. Logs need to be analyzed periodically to ensure system security, and they also need to be analyzed to prepare usage reports ‘and other basic system data. In edation, log files need to be rotated periodically; log files grow regularly overtime, and must be timmed {to prevent total disk space consumption. However, most daemons. ‘complain f their log files are rotated while the daemons are running ‘and have the logfiles open for writing, s0 rotation of log files normally requires that daemons be stopped, logfiles be moved, new log files be created, and then daemons be restated ‘As might be apparent, both log analysis and log rotation are tedious, repetitive, time-consuming tasks. Linux provides utlites which can help with both of these chores. The lagrotate program can be used 10 automate log file rotation, while the logwateh program can Perform basic log file monitoring and analysis. logrotate ‘The system ships with logrotate configured to manage system logs. Basie logrotate configuration is done in the /ete/ logrotate. cont file In this fle, general options can be set tke how frequently log files should be rotated, how many old logfiles should be kept, and whether or not old logfiles should be compressed. in edition, all the Configuration files in /ete/ togrotate.d/ are also interpreted by logrotate. Files in this directory configure logrotate to manage spectic services, by teling it information like what logfiles generated 146 Log Management Nar/os) * Standard location fr lg les egeotate * Log rotation wtlty + Allows automatic rotation, compression, removal and maling of {og fos + Funs diy om /etc/cron.daily/ logrotate 1 fete/ logrotate cont ‘Main contigration fe + fete/ logrotate. df ‘Alas In this deter uted fr eonfiguetin x¢ wal by that service need to be rotated. how the service should be restated i it neds to be restarted after lg fie rotation, and so forth, ‘The /ete/togrotate.d/ directory makes it easy for RPMs to install {and remeve configuration files on the system specifying how the log fles produced by the software In thet RPM should be managed. “Main logrotate Configuration File The /ete/Logrotate.cont flo mainly acts to set defaults and then Inchudes daemon specitic configuration les Irom the Jete/ logrotate. dre y rotate ¢ lereate include /ete/togrotate.d /varNoghweap monthly rinsize 1M create 8664 root utmp rotate 1 ARR RR RRR RAR RRR RAR AAA RAR AAAww eww wwwwwewwwwwYuUYUwuwwewowwuws fogwatch Red Hat Enterprise Linux also ships Kirk Bauer's Loguatch package. logeatch isan easily customized suite of software for analyzing, system logs to ently interesting messages. Logvatch can analyze {og fils from most popular services, and it can be configured to do 0 automatically on 2 gular basis, emaling the administrator any. ‘results. Commonly it is Configured to monitor log flles on an hourly ‘oF nightly basis for suspicious activity on the system. ‘The default Red Hat Enterprise Linux configuration runs Logvatch ‘each night trom /etc/eron.daily/ and emails a report to root. Log Anomaly Detector Day-to-day Information og messages considered unimportant > Foponsyects upon anomalous massages 107Task 1: Setting up a Full Debug Logfile Page: 149 Time: 5 mines Requirements: @ (1 station) ‘Task 2: Remote Syslog Configuration Page: 14-11 Time: 15 mirutes Requirements: (2 stations) 148 ARR RAR RRR RRR RR ARR RRA AA AMARAwwwwewwwwwwrwwwwwwvwwwwwowwewvwvEd Ararat ep 1 14 k ‘ask 1 Requirements Setting up a Full Debug: {1 station) Logfile selevnca etnated Tener Sma v 2 3) 4) 5) ‘Sotting up a debug log fle cn help troubleshoot system issues The following actions require administrative privileges. Switch to 8 root login shell $ su -t Password: makeitso =] Configure Reysiog to log all debug messages tothe file /var/log/debug. Add the Tollowing lines to the botiom of /ete/rsyslog. cont: Tete/reysiog. cont Reload the syslog daemon so the changes go into effect 4 service reystog reload ++ output omitted... ‘Test the syslog configuration by sending @ test log message of info priority: 4 Logger -p info -t info-lvl-nsg “This is an info-priority message" CCheck to see which file the test message was logged to: # tail -n 2 /var/tog/ {debug messages) Ivar/Nog/debug <== “The inf poity messages goto both og ls. reyslogd: (origin softvare="rsyslogd"6.2" x-pid="1269" x-infor"http://ww.rsyslog.con"] (restart inforlvi-nsg: This is an info-priority message = /var/tog/nessages: <= 149syslogd! [origin softvares*rsyslogd"6.2" x-pid="1269" x-infosrhttp://inw.rsyslog.con) (xe)start info-ivl-nsg: This is an info-priority message 66) Test the syslog configuration by sending atest log message of debug priority: 4 logger ~p debug -t debug-Lvi-nsg “This is a debug-priority message" 7) Check to see which fila the test message was logged to Ivar/og/debug <= inforlvi-nsg: This is an info-priority message debug-ivl-nsg: This is an debos-priority message > /var/og/nessages <= ‘Noize tat hare was no deb pty mssege logos reyslogé: (origin softvare="rsyslogd* > swersion="4.6.2" x-pid-"1269" x-info=*http://way.tsyslog. com") (re)start inforlvi-asg: This is an info-priority message Cleanup 8) Configure Rsysog to stop logging debug messages. Remove the following lines from the bottom of /etc/rsystog.conf: '9) Reload the daemon so the changes go into effect 4 service rsyslog reload +e output onitted 10) Administrative privileges are no longer required; exit the root shell to etuin to an ‘unprivleged account f exit 1410 POR ROR ERA RRR ARR ARAN AAR A NyPOW EVwwUYEwwwVUBUTVVUVWUWWVoWws ‘Objectives Lab 14 Mere Rayon tn on tena for emo og massages ‘onfgure Rayos to sond eesongus to svete owang sos Task 2 Aue Remote Syslog Configuration ee ‘Estimated Tine: 16 minutos neteronce Se lleva 1) Find out what port is used for remote Rsysiog logging: § grep ~i systog /ete/services 'S14/odp 2), The following actions requte administrative privileges. Switch to a zoot login shell Ssu-t Password: makedtso =] 3) Determine whether or not Rsystog is istening on the network: # netstat -taupen | grep 514 No cutut bare shows us that slg isnot caenty Istering on the network 4) Configute Rsysiog to listen on the network for remote log messages (port §14 by default (Fle: Tete/esystog cont T Provides Wor syslog reception «| ssuadtaad impose ~ | ascoz6evortu S14 + sodvoad imedp.so + |Suppservornun 514 1a5) Start the syslog so that changes go into effect: § service rsystog retosd Reloading eysten Logger... (on) 6) Determine whether or not the daemon is listening for remote network onnections: 4 avtstat -eaupen | grep Sit tp 88 6.8.0.0 . udp 8 6: 3 6 39887 2363/rsystogd 39888 2363/zeyeloga 7) Wait for your lab partner to reach this point before continuing. 8) Configure Reyslog to send messages to your partner's system: (Fie: Fete/raystog.cont ¥ tog anything (except mall) of level Info or Righer- # pont log private authentication messages! + |+sinfo;mait -nonejauthpriv.none;cron.none 9) Reload the daemon so thatthe changes go into effect: F service reystog reload Reloading system logger... (on) 10) Send a test log message: ¥ logger -p info ~t info-tvi-nsg “Message fron station!” 11) ‘ait fr your lab partner to reach tis point before continuing. 12) Observe the contents of the system log: 1 tail -n2 /var/Log/nessages 412 RRR RRR RRR RAR RAR AR AAA AMARA AAA AAGVY CWO EwOUOWWUUwYUwUYeUUowWwUUWWd 38 station® rsyslogd: (origin softwares"rsyslogd" > + This ina, and the net is the message which was "4.6.2" xepid="1278" acinfom"http://n.rsyslog.con"T> creed an you prs system and forwarded to (re)stare ‘ows Jan 12 15:39:05 stationY info-twl-nsg: Message from station Cleanup: 13) Configure Rsysiog to stop listening on the network for remote log messages: [Tete | Provides UDF ayslog reception - | stodtoad-imadprse * | suppeeeveetunSi4 +] fswodtoad imidp.s0 +] ésuneservertun 514 14) Contigure Rsysiog to stop sending messages to your partner's systern Fle: Tete/rsysiog.cont Fog anything (except mail) OF Tevel info or Righer- | on’t. log private authentication messages! -info;nail.none;authpriv.none;cron.none var/tog/aessages 15) Restart the daemon so that the changes go into effet £ service reystog reload Shutting dovn systea logger: {ox } Starting systex logger: (OK) 16) Administrative privileges are no longer required: ext the root shell to return to an Lnprivleged account exit 1413