Log Rotation and Management: Preventing Disk Space Issues on Web Servers
Disk space filling up on a web server is one of those problems that can catch you off guard. One day your site runs fine, the next you are dealing with a server that will not write logs, services that refuse to start, or an application that crashes with mysterious errors. The root cause is often simpler than it appears: log files have grown unchecked, consuming every available byte on the disk.
Log rotation is the standard practice of managing log files so they do not grow indefinitely. Instead of allowing a single log file to reach tens of gigabytes, log rotation breaks logs into manageable chunks, compresses older entries, and removes files that are no longer needed. This keeps your server healthy and makes troubleshooting far easier when something does go wrong.
Why Log Files Grow Out of Control
Every service running on a Linux server generates logs by default. Your web server writes access logs for every request. Your database logs queries, connections, and errors. System services log kernel messages, authentication attempts, and application output. On a busy server, these files can grow by megabytes every minute.
Without rotation, you end up with files that are difficult to open, slow to read, and impossible to back up efficiently. A single access.log file can reach 50GB on a high-traffic site. When that happens, tools like grep struggle, log analysis becomes impractical, and disk space disappears faster than you expect.
There is also a security dimension. Large log files make it harder to spot suspicious activity. An attacker who knows your logging setup might try to fill a log file deliberately as part of a denial-of-service attempt. Proper log management reduces that attack surface.
How Logrotate Handles Log File Management
On most Linux distributions, logrotate is the tool that handles this automatically. It runs daily via cron and checks configuration files that tell it which logs to rotate, how often, and what to do with older copies.
The basic concept works like this. When logrotate determines that a log file needs rotating, it renames the current file by adding a date or sequence number. For example, /var/log/nginx/access.log becomes /var/log/nginx/access.log.1. A new empty file takes the place of the original. Logrotate can then compress the old file, remove files older than a set number of rotations, and notify the service so it starts writing to the new file.
Configuration lives in /etc/logrotate.conf for global settings and /etc/logrotate.d/ for service-specific rules. Most packages that install a service add their own configuration file here automatically.
Understanding the Logrotate Configuration Structure
A basic logrotate configuration looks straightforward but has several important options that affect how well it works in practice.
/var/log/nginx/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
create 0640 www-data adm
sharedscripts
postrotate
invoke-rc.d nginx rotate-nr欠 --quiet >/dev/null 2>&1 || true
endscript
}Each directive controls a specific behaviour. Understanding what each one does helps you configure rotation correctly for your server's workload.
Rotation Frequency: Daily, Weekly, or Monthly
The daily directive tells logrotate to check and rotate logs once every day. This works well for most web servers. If your site has moderate traffic, daily rotation keeps files manageable in size. weekly rotation is suitable for low-traffic servers where logs do not grow quickly. monthly is rarely useful for production web servers but might apply to systems with very low activity.
Choosing the right interval depends on how quickly your logs grow. A busy e-commerce site might need daily rotation with a high retention count. A small brochure site might be fine with weekly rotation and fewer saved copies.
Retention Count and Disk Space
The rotate 14 directive tells logrotate how many old log files to keep. In this example, fourteen rotated files are kept, which means roughly two weeks of logs at daily rotation. Adjust this number based on how long you need to keep logs for debugging, compliance, or reporting purposes.
More rotations mean more disk space used by old logs. If disk space is tight, lower the rotation count or increase the rotation frequency. If you need longer history, increase the count but monitor your disk usage to make sure you have enough space.
Compression Reduces Storage Needs
The compress directive compresses rotated log files using gzip by default. This can save significant space, especially for text-based logs that compress well. A 1GB access log might compress to under 100MB.
The delaycompress directive is worth using when your service does not immediately close its file handle after rotation. It delays compression by one cycle, giving the service time to finish writing to the rotated file. Without it, compression might fail on files that are still being written to.
Handling Missing and Empty Files
The missingok directive tells logrotate to continue without error if a log file does not exist. This is useful for servers where certain logs might not exist depending on configuration.
The notifempty directive skips rotation for files that are empty. Rotating an empty file creates unnecessary noise in your logs and can cause services to restart unnecessarily. This is almost always the right behaviour for production servers.
File Permissions and Ownership
The create directive sets the permissions, owner, and group for the new log file created after rotation. In the example, 0640 www-data adm means the new file gets read and write for the owner, read for the group, and nothing for others, owned by the www-data user and adm group. Matching the original file's ownership keeps your web server happy and maintains appropriate security.
Common Logrotate Configuration Mistakes
Even experienced server administrators make mistakes with logrotate configuration. Some of these cause problems immediately, while others silently fail to protect your disk space.
Forgetting to Notify Services After Rotation
When a log file is renamed, the service that is writing to it continues writing to the old file handle. Unless you restart the service or send it a signal, it keeps filling the renamed file. The postrotate and endscript block handles this by running a command after rotation completes.
For Nginx, sending a USR1 signal tells the worker processes to reopen their log files:
postrotate
kill -USR1 $(cat /var/run/nginx.pid)
endscriptFor Apache, the command differs slightly:
postrotate
/usr/sbin/apache2ctl graceful >/dev/null 2>&1 || true
endscriptSkipping this step means your service writes to rotated files indefinitely. The original log file shrinks to nothing while a renamed copy grows to fill your disk.
Setting Rotation Intervals That Do Not Match Your Traffic
Using weekly rotation for a high-traffic server can result in massive log files that are difficult to manage. If your access logs grow by several gigabytes per week, weekly rotation means each file could be several gigabytes. Daily rotation keeps files smaller and easier to handle, even if you keep more total copies.
Test your configuration by monitoring log file sizes over time. If you notice files consistently exceeding a few hundred megabytes before rotation, consider increasing the frequency.
Not Monitoring Disk Space After Configuration Changes
Changing rotation settings affects how quickly old logs accumulate. Increasing the rotation count means keeping more copies. Decreasing compression might free CPU but use more disk space. Always verify your disk usage after changing logrotate configuration to make sure you are not creating new problems.
Log Rotation for Different Services
Each service on your server has its own logging patterns and requirements. While the logrotate mechanism is the same, the configuration details vary.
Nginx Access and Error Logs
Nginx typically generates two main log files: access.log for every request and error.log for errors and warnings. High-traffic sites can generate gigabytes of access logs daily. A sensible configuration rotates access logs daily, keeps fourteen days, and compresses older copies. Error logs can rotate less frequently since they grow more slowly, but keeping them for at least thirty days helps with diagnosing intermittent problems.
MySQL and MariaDB Logs
Database servers log queries, connections, and errors differently depending on configuration. Binary logs for replication need separate handling and are not typically rotated with logrotate. General query logs can grow very quickly on busy databases. In most production environments, general query logging is disabled and slow query logging is used instead with a shorter retention period.
Application Logs
If your web application writes its own logs, those files need rotation too. PHP applications might write to a file defined in php.ini or a custom location. Node.js applications often write to pm2 logs or custom log files. Check where your application writes logs and add appropriate logrotate rules.
Automating and Testing Logrotate
Logrotate runs automatically from /etc/cron.daily/logrotate on most systems. You do not need to trigger it manually in normal operation, but knowing how to test your configuration is valuable.
To test a configuration without actually rotating files, use the debug flag:
logrotate -d /etc/logrotate.d/nginxThe -d flag shows what logrotate would do without making any changes. This lets you verify your directives are parsed correctly and see which files would be rotated.
To force rotation outside the normal schedule for testing:
logrotate -f /etc/logrotate.d/nginxThe -f flag forces rotation regardless of the schedule. Use this only for testing, not in production unless you have a specific reason.
Monitoring Log File Growth
Configuration alone is not enough. You need to monitor disk usage to catch problems early. Setting up basic monitoring helps you spot when logs are growing faster than expected or when rotation is not working correctly.
Regular monitoring also helps you tune your rotation settings over time. A server that starts with moderate traffic might grow significantly, requiring more frequent rotation or shorter retention periods.
For practical guidance on setting up server monitoring, see my article on Linux server monitoring with htop and Netdata. That guide covers how to track disk usage and resource consumption alongside other server health metrics.
When Logs Indicate a Problem
Large or growing log files are not just a storage issue. They often signal something worth investigating. An access log that suddenly doubles in size might indicate unexpected traffic, a misconfigured application, or an attempted attack. An error log that grows rapidly often points to application bugs, configuration problems, or service degradation.
Reviewing logs regularly, even briefly, helps you catch issues before they become serious. Log rotation makes this practical by keeping files at a manageable size and retaining enough history to spot patterns.
Backup Considerations for Log Files
Rotated log files contain valuable information that might be needed for debugging or compliance. When planning backups, include rotated logs in your strategy. Compressed logs are smaller and back up efficiently, but they do need to be retained long enough to be useful.
Consider what information you actually need from logs. Access logs are useful for traffic analysis and debugging. Error logs help diagnose problems. Application logs might contain sensitive data that requires careful handling under data protection rules. Only keep logs as long as you genuinely need them, and make sure your retention policy complies with any regulations that apply to your situation.
Related practical reading
These related guides can help you connect this topic with the wider website, server, security, and support decisions around it.
- GraphQL in PHP vs REST: When GraphQL Is the Better Choice - useful background for related development decisions
Keeping Your Server Healthy with Proper Log Management
Log rotation is one of those maintenance tasks that is easy to overlook until it causes problems. Setting up sensible rotation policies, testing your configuration, and monitoring disk usage keeps your server running smoothly and makes debugging much easier when issues arise.
Most Linux servers already have logrotate installed and configured for standard services. Reviewing those defaults and adjusting them to match your traffic patterns and storage capacity is a straightforward step that pays off over time.
If you are unsure whether your current setup is configured correctly, or if you have inherited a server without documentation of its log management approach, a quick review of your logrotate configuration and disk usage can reveal whether adjustments are needed.