How to setup log rotation and Webalizer for Apache logs

Sep 15, 2012

These instructions are based on my system using Centos 6.3 but should apply pretty generally.

For my CentOS 6.3 system logrotate is already installed and inclued in cron.daily to run every day.

In CentOS packages add log rotation configuration files to /etc/logrotate.d .

There is also a primary log rotation configuration at /etc/logrotate.conf .

If you are using a different distribution or do not have logrotate installed already you will have some extra steps to set that up.

These are some of the values in the configuration along with their defaults in CentOS 6.3

  • How frequentyly you want your log files rotated(default weekly)
  • How many prior logs you keep(default 4)
  • Whether compressed logs are enabled (default off)
  • Whether an empty file is created after rotation

To begin, backup all the default configurations so you can go back if you make a mistake. 

mkdir ~/logrotate_bak  

cp /etc/logrotate.conf ~/logrotate_bak  

cp -R /etc/logrotate.d ~/logrotate_bak`

This is the default /etc/logrotate.d/httpd file from my system.


/var/log/httpd/\*log {
    missingok
    notifempty
    sharedscripts
    delaycompress
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

The configuration for Apache httpd assumes that logs all match the pattern

/var/log/httpd/*log

This is not the case for me, so I change this to:


/srv/www/example.com/logs/\*log {
    missingok
    notifempty
    sharedscripts
    delaycompress
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

Also, I want to rotate my logs daily and keep 60 days instead of 4 weeks,

so I replace this:


# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

With this:


# rotate log files daily
daily

# keep 60 days of backlogs
rotate 60

You can also specify to rotate the log files when they get to a certain size.

CentOS default already includes the dateext option which includes the date in the old log file’s name, so we want to keep that.


dateext

We also want to turn on the compress option to save space. Old logs will be compressed with gzip.


compress

So all in all it looks like this now:


# rotate log files daily
daily

# keep 60 days of backlogs
rotate 60

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
        minsize 1M
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.

We are getting closer, but now we need to tie in Webalizer, but first of course we need to install it.

yum install webalizer

We do not want to use the default cron script for Webalizer so remove that.

rm /etc/cron.daily/00webalizer

We need to setup example directories for each domain. Assuming that /srv/www/example.com/public_html is your document root for your domain.

mkdir -p /srv/www/example.com/public_html/stats

We want to place some access control on the Webalizer output so not everyone can view all of our statistics.

Create an .htaccess file in the stats directory for your domain. Or alternately you can configure it directly into your httpd.conf or a config file included by your httpd.conf.


# Place this in /srv/www/example.com/public\_html/stats/.htaccess
AuthType Basic
AuthName "Restricted Files"
# (Following line optional)
AuthBasicProvider file
# This file MUST not be underneath your document root.
AuthUserFile /srv/www/example.com/passwd
# Allow anyone added to the passwd file who has their correct password.
Require valid-user

Add users to the password file like so:

htpasswd /srv/www/example.com/passwd mylogin

Also note that Apache httpd.conf files often have “AllowOverride None” directives in them which will make Apache purposefully ignore your .htaccess files. To insure it works, you may need to add something like this to your VirtualHost block for your domain.


 # let .htaccess work

 AllowOverride all

If you are going to process multiple virtual domains seperately, then you will want to adjust Webalizer’s configuration file, typically in /etc/webalizer.conf, so that it does not keep the incremental data in the same location across domains. To do this change the value of the IncrementalName setting so that it is a relative path and not an absolute path which will cause Webalizer to store it inside the output directory you specify on the command line below.

Test Webalizer directly from the command line before enabling it.

webalizer -n example.com -o /srv/www/example.com/public_html/stats /srv/www/example.com/logs/access.log

Check the output at example.com/stats then when satisifed delete it so it does not cause data to be double processed later on.

rm -Rf /srv/www/example.com/public_html/stats/*

Then we need to update our log rotation settings for httpd so that a prerotate step runs webalizer before thelog file gets compressed by using the prerotate section and the delaycompress option

If you want to handle multiple domains, you will want to write a script to run from logrotate that handles all of them. If you are just handling a single domain then you can put the webalizer command directly into the logrotate configuration file for httpd, like so:


/srv/www/example.com/logs/\*log {
    missingok
    notifempty
    sharedscripts
    delaycompress
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
    prerotate
        /usr/bin/webalizer -n example.com -o /srv/www/example.com/public\_html/stats /srv/www/example.com/logs/access.log
    endscript
}

The sharedscripts line is important, it means prerotate and postrotate scripts are run once only for all files instead of once for each file that matches the pattern.

Here is how you can make a script for running webalizer on all domains. It assumes that you have a directory for each domain underneath /srv/www named the same as the domain itself.

mkdir -p /opt/bin  

touch /opt/bin/webalizerAll.sh  

chmod +x /opt/bin/webalizerAll.sh`

Edit /opt/bin/webalizerAll.sh with your favorite editor( reads VIM for me) and set it up like so.

#!/bin/sh

DOMAINS=`ls /srv/www`
for DOMAIN in $DOMAINS; do
    echo "Processing $DOMAIN"
    /usr/bin/webalizer -n $DOMAIN -o /srv/www/$DOMAIN/public\_html/stats /srv/www/$DOMAIN/logs/access.log
done

Then replace the prerotate command with 

/opt/bin/webalizerAll.sh

There is more information on this at the FAQ for Webalizer at http://www.webalizer.org/faq.html .

And now you can customize your Webalizer configuration as desired.

References

  • Online version of logrotate man page: http://linuxcommand.org/man_pages/logrotate8.html
  • Webalizer home page: http://www.webalizer.org
  • Apache web server: http://projects.apache.org/projects/http_server.html
  • CentOS: http://www.centos.org/
  • Apache web server authentication howto: http://httpd.apache.org/docs/2.2/howto/auth.html

Copyright 2002-2024 Richard Anton. All rights reserved.