Locate and interpret system log files

The default directory for system logs is /var/log.  Some processes/daemons/applications will log to a file in this directory, others to a file inside a directory. An example of the former is the kernel or ssh daemon (sshd), /var/log/dmesg and /var/log/secure respectively while an example of the former is SELinux that logs to /var/log/audit/audit.log

It is possible to change this location for some applications, e.g. mysql, therefore if you can't find a log file in /var/log, you need to have a look at the configuration file for the application to find out where it's logging file actually is.

I normally use tail -n 50 /var/log/logfile, this will display the last 50 lines of  the log file, I find that the default 10 lines is rarely enough. I sometimes use less /var/log/logfile as this provides vi style search capabilities, which can be quite handy. Obviously, you can use vi, cat or even emacs.

The -f flag for tail is very useful as it will essentially let you watch the log file in real time, so for instance tail -f /var/log/secure will let you watch what the ssh daemon is doing.

If instead of the end of the file, you want to have a look at the beginning of the file, then head is your command.

This is as far as it goes locating and reading logs, the interpretation part of this objective is best learnt by practice. The logs might seem daunting at first, but they tend to be reasonably self explanatory, although you might need to increase the log level before they start making sense. Sample sshd log below.

Jun  3 12:24:28 Subversion sshd[21785]: Accepted password for user from 10.168.20.227 port 35613 ssh2
Jun  3 12:24:28 Subversion sshd[21785]: pam_unix(sshd:session): session opened for user user by (uid=0)

The above lines represent a successful login attempt with password to a server called Subversion from an ip address 10.168.20.227. The ssh daemon is running with process id 21785 for this session and the connection was established on port 35613 using ssh2. pam_unix has successfully authenticated the user user and has opened a session for it.

A special mention should be made of the SELinux logs. A lot of the time you will find yourself (at least I have) asking why something is not working, e.g. why is iptables-save not working properly?, and the answer could well be in the audit logs.

type=AVC msg=audit(1307099179.414:42289): avc:  denied  { write } for  pid=21674 comm="iptables-save" path="/tmp/iptables.txt" dev=sda3 ino=11171 scontext=unconfined_u:unconfined_r:iptables_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file

The key parts are in bold, namely what happened (write denied) and what were you trying to do (run iptables-save), other issues might be at hand too. I'll show how to get rid of SELinux issues like these in the SELinux objectives post.