Split Apache Log Files
It provides the configuration required to split the log files generated by the Apache Web Server.
This tutorial provides the configurations required to configure the virtual host of Apache Web Server to generate the custom logs and also split the logs according to date. It also provides the configuration to specify the size of the log files generated by the Apache Web Server.
It provides examples of the rotatelogs program. At the end of this tutorial, it also shows the configurations required by the vlogger to split the logs. Also, I have used Ubuntu 20.04 LTS while writing this tutorial. The steps should be the same on other versions of Ubuntu and Linux systems.
Log Formats
The pre-defined log formats provided by the Apache Web Server are shown below. Similar to Apache, we can also specify our own log formats.
# Common Log Format
# Logs - remote host IP address, authenticated user ID, date and time, request, response status code
LogFormat "%h %l %u %t \"%r\" %>s %b" common
# Combined Log Format
# Additionally Logs - refererrer and user agent
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
Virtual Host - Using Main Log File
The default virtual host of the Apache Web Server should be similar to the virtual host as shown below.
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
ServerAdmin admin@example.com
DocumentRoot /var/www/example.com/html
<Directory /var/www/example.com/html>
Options -Indexes +FollowSymLinks
DirectoryIndex index.php
AllowOverride All
Require all granted
</Directory>
</VirtualHost>
This virtual host does not specify the log files path, hence it uses the main server log file of the Apache Web Server. This is sometimes problematic in the scenario where the same Apache server is used to host multiple websites. The logs of all the websites go to the main server log.
Virtual Host - Own Log File
The default virtual host of the Apache Web Server should be similar to the virtual host as shown below.
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
ServerAdmin admin@example.com
DocumentRoot /var/www/example.com/html
<Directory /var/www/example.com/html>
Options -Indexes +FollowSymLinks
DirectoryIndex index.php
AllowOverride All
Require all granted
</Directory>
ErrorLog /var/www/example.com/logs/error.log
CustomLog /var/www/example.com/logs/access.log combined
</VirtualHost>
This virtual host specifies the error and access log files path hence uses its own files to generate the logs.
Split Log Files
On the websites having moderate to high traffic, the size of the log files starts increasing as the time flows. This makes it difficult to open and analyze the large log files. We can configure the logger to split the log files using the rotatelogs function according to the date as shown below.
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
ServerAdmin admin@example.com
DocumentRoot /var/www/example.com/html
<Directory /var/www/example.com/html>
Options -Indexes +FollowSymLinks
DirectoryIndex index.php
AllowOverride All
Require all granted
</Directory>
ErrorLog /var/www/example.com/logs/error.log
CustomLog "|/usr/bin/rotatelogs /var/www/example.com/logs/access_%Y.%m.%d.log" combined
</VirtualHost>
If you are sure that your log files can still grow in size in a single day, we can further split the log file by specifying the maximum size of a single file as shown below. It will generate multiple files for the same date.
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
ServerAdmin admin@example.com
DocumentRoot /var/www/example.com/html
<Directory /var/www/example.com/html>
Options -Indexes +FollowSymLinks
DirectoryIndex index.php
AllowOverride All
Require all granted
</Directory>
ErrorLog /var/www/example.com/logs/error.log
CustomLog "|/usr/bin/rotatelogs /var/www/example.com/logs/access_%Y.%m.%d.log 10M" combined
</VirtualHost>
The other options to split the log files are shown below.
# Split the log by rotation time - 24 hrs - 86400
CustomLog "|/usr/bin/rotatelogs /var/www/example.com/logs/access.log 86400" combined
# Split the log by date and switch to new file every day at midnight
CustomLog "|/usr/bin/rotatelogs /var/www/example.com/logs/access_.%Y.%m.%d.log 86400" combined
# Split the log by file size - 10M
CustomLog "|/usr/bin/rotatelogs /var/www/example.com/logs/access.log 10M" combined
We can also avoid the log file growing too big by truncating it at server startup and further truncate it once in a day as shown below.
# Truncate the log by file
CustomLog "|bin/rotatelogs -t /var/www/example.com/logs/access.log 86400" combined
This makes sure that there is only one log file and it gets truncated on server startup and once in a day. It's useful in the scenarios where we are using 3rd party tools like logstash to export the logs.
Split logs using vlogger
Apart from the rotatelogs program, we can also use vlogger to split the logs. We can easily split the logs according to the virtual hosts without updating the virtual host configuration.
Install vlogger as shown below.
# Install vlogger
sudo apt install vlogger
Now update the main configuration file of the Apache Web Server as shown below.
# Update Apache Configuration
sudo nano /etc/apache2/apache2.conf
# Update the log formats
#LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%v %h %l %u %t \"%r\" %>s %b" common
#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
# Add CustomLog directive below the LogFormat
CustomLog "| /usr/sbin/vlogger -s access.log /var/log/apache2" combined
# Save and exit the nano editor
# Restart Apache Web Server
sudo systemctl restart apache2
This is the only configuration required to split the log files according to the virtual hosts and date. Now a separate log file will be generated for each virtual host at /var/log/apache2/<virtual host>. The log file path should be similar to /var/log/apache2/example.com/07062020-access.log and /var/log/apache2/example.com/07062020-error.log.
Similar to rotatelogs, we can also specify the size(in bytes) of the log files generated by vlogger using the -r option as shown below.
# CustomLog directive
CustomLog "| /usr/sbin/vlogger -s access.log -r 10000000 /var/log/apache2" combined
Summary
This tutorial provided the configurations to configure the virtual host to specify the log files path and to split the log file by the date and size.
