The FreeBSD Diary

Every decent system generates logs. They are useful both from a forensic and from a debug point point of view. Some systems generate huge volumes of logs. Scanning those logs manually is both tedious and error-prone. This calls for an automated solution. Enter logcheck. Logcheck will scan your log files and report any entries which do not match a list previously flagged as OK to ignore. The pattern matching is flexible and easily extended.

logcheck has been around at least 10 years. I starting using logcheck in 1999, just about 10 years ago. Since then, logcheck underwent quite a transformation. It once had just a handful of matching files. Now it has over 180 files.

logcheck works by ignoring known benign patterns and reports any log file entries that do not match those patterns. You can add to these patterns easily.

Logcheck can scan a number of files. The list is kept in /usr/local/etc/logcheck/logcheck.logfiles. I choose to scan these files:

# these files will be checked by logcheck
# This has been tuned towards a default syslog install
/var/log/messages
/var/log/auth.log
/var/log/maillog

NOTE: the comments are not mine.

For logcheck to scan all the files on a default FreeBSD system, you will need to make some changes to file permissions, /etc/newsyslog.conf, and /etc/group. See the next section for details.

logcheck runs as the logcheck user:

# grep logcheck /etc/passwd
logcheck:*:915:915:Logcheck system account:/var/db/logcheck:/usr/local/bin/bash

cd /usr/ports/security/logcheck
make install clean

If the cd fails, you need to do this first because you probably don't have a ports tree checked out:

portsnap fetch && portsnap extract

If you do not alter the permission and update some configuration files, you'll soon get one of these emails:

To: root@ngaio.example.org
Subject: Logcheck: ngaio.example.org 2009-11-20 12:02 exiting due to errors
Message-Id: <20091120120201.7ACFF17104@ngaio.example.org>
Date: Fri, 20 Nov 2009 12:02:01 +0000 (GMT)
From: logcheck@ngaio.example.org (Logcheck system account)

Warning: If you are seeing this message, your log files may not have been
checked!

Details:
Could not run logtail or save output

Check temporary directory: /tmp/logcheck.ZOjfJO

Also verify that the logcheck user can read all files referenced in
/etc/logcheck/logcheck.logfiles!

declare -x HOME="/var/db/logcheck"
declare -x LOGNAME="logcheck"
declare -x MAILTO="root"
declare -x OLDPWD
declare -x PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
declare -x PWD="/var/db/logcheck"
declare -x SHELL="/bin/sh"
declare -x SHLVL="1"
declare -x USER="logcheck"

The email contains the wrong location for the file. It is assuming an installation location which has been changed at configuration/install time and /usr/local/sbin/logcheck has not been refreshed accordingly. I have submitted a patch for that. Check the permissions for the files listed in /usr/local/etc/logcheck/logcheck.logfiles:

# ls -l /var/log/messages /var/log/auth.log /var/log/maillog
-rw-------  1 root  wheel   6564 Nov 28 21:13 /var/log/auth.log
-rw-r-----  1 root  wheel     60 Nov 28 00:00 /var/log/maillog
-rw-r--r--  1 root  wheel  83127 Nov 28 22:00 /var/log/messages

As you can see, the logcheck user will be unable to read auth.log and maillog. We can change that.

# chgrp logcheck /var/log/auth.log /var/log/maillog
# chmod g+r /var/log/auth.log
# ls -l /var/log/messages /var/log/auth.log /var/log/maillog
-rw-r-----  1 root  logcheck   6564 Nov 28 21:13 /var/log/auth.log
-rw-r-----  1 root  logcheck     60 Nov 28 00:00 /var/log/maillog
-rw-r--r--  1 root  wheel     83277 Nov 28 22:05 /var/log/messages

logcheck will now be able to read the files, but as you know, these files are rotated by newsyslog.conf. So let's see the entries for them:

# egrep "/var/log/auth.log|/var/log/maillog" /etc/newsyslog.conf
/var/log/auth.log                       600  7     100  *     JC
/var/log/maillog                        640  7     *    @T00  JC

The above is before my changes, the following is after:

# egrep "/var/log/auth.log|/var/log/maillog" /etc/newsyslog.conf
/var/log/auth.log       root:logcheck   640  7     100  *     JC
/var/log/maillog        root:logcheck   640  7     *    @T00  JC

Note that you have to add the root:logcheck to both *and* change the mode for auth.log to 640.

Recent versions of logcheck default the outgoing email to the logcheck user. To get these emails sent to myself, I added this entry to /etc/mail/aliases:

logcheck:       dan

logcheck will initally produce notices about things you do not care to see again. They are normal for your system and they do not need to be brought to your attention again. You can train logcheck to ignore these items. You will see both System Events and Security Events emails. For example:

Security Events
=-=-=-=-=-=-=-=
Nov 28 16:37:55 dbclone postgres[93778]: [2-1] ERROR:  table "mac" does not exist

System Events
=-=-=-=-=-=-=
Nov 28 16:28:22 dbclone bacula-dir: Shutting down Bacula service: localhost-dir ...

These items are normal for this system. It is used for Bacula regression testing. For the Security Events, I created /usr/local/etc/logcheck/violations.ignore.d/local-postgres with the following contents:

# grep mac /usr/local/etc/logcheck/violations.ignore.d/local-postgres
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postgres\[[0-9]+\]: \[[0-9]+\-[0-9]+\] ERROR:  table "mac" does not exist

That preamble seems like a lot. But I grabbed it from logcheck-postgres. The logcheck project recommends that you put your own customizations into files prefixed with local- so they are easily identified. logcheck itself does not care.

For the System Event, I added this entry to /usr/local/etc/logcheck/ignore.d.server/local-postgres

bacula-fd: Shutting down Bacula service: localhost-fd

Notice that my System Event exceptions are specified in the ignore.d.server directory. This is because I selected the following option in /usr/local/etc/logcheck/logcheck.conf:

REPORTLEVEL="server"

If you are using "workstation", you would add your file to the ignore.d.workstation directory.

There you go. That should get you started with logcheck. I've been using it for 10 years. It's a great idea. I hope and trust it will save you a great deal of ready. Best wishes.