Why Server Logs Are the First Place to Look When Something Breaks

When a website goes down, an application behaves unexpectedly, or a server starts running slowly, server logs are where the answers are. Before you start changing configurations, restarting services, or escalating to support, spending ten minutes with the logs tells you what actually happened. This guide covers which logs to check, what to look for, and how to interpret what you find.

Where Linux Stores Its Logs

On most Linux distributions, system logs live in /var/log/. This directory contains logs from the kernel, system services, applications, and the web server. Understanding what each log file contains narrows down where to look when an issue occurs.

ls -lh /var/log/

total 2.1G
drwxr-xr-x  2 root root 4.0K May 20 10:00 apache2/
drwxr-xr-x  1 root root 4.0K May 20 09:45 auth.log
drwxr-xr-x  1 root root  124M May 20 00:00 kern.log
drwxr-xr-x  1 root root  2.1M May 20 10:00 nginx/
-rw-rw-r--  1 root root  89M May 20 00:00 dpkg.log
-rw-r-----. 1 root syslog  89K May 20 09:45 syslog

The auth.log file records every authentication attempt, sudo usage, and login event on the server. The syslog and kern.log contain system-level events from various services. The web server directories contain access logs and error logs, which are often the most useful when debugging web applications. If you are running multiple services on a single server, familiarising yourself with this directory structure saves significant time during incident response.

Reading Apache Error Logs

Apache stores error logs in /var/log/apache2/error.log by default, or in a custom location if the virtual host configuration specifies a different path. The location defined in your Apache configuration takes precedence over the default.

tail -50 /var/log/apache2/error.log

The tail command shows the last 50 lines of the log file. As issues occur, new entries are appended to the end of the file. Watching the log in real time is often more useful than reading a static snapshot, particularly when you are actively reproducing an issue.

tail -f /var/log/apache2/error.log

The -f flag follows the file, printing new lines as they are written. Leave this running in one terminal, reproduce the issue in another, and watch the error messages appear in real time.

A typical Apache error log entry looks like this:

[Wed May 20 09:45:23.234567] [core:error] [pid 12345:tid 140234567890] 
[client 192.168.1.60:54321] AH00124: Request exceeded the limit of 10 
internal redirects due to probable configuration error. 
Rules, referer: https://example.com/page

This tells you the timestamp, the error severity level, the process ID, the client IP address, and the error description. The Request exceeded the limit of 10 internal redirects message indicates a rewrite loop, usually caused by a misconfigured .htaccess file or conflicting redirect rules. Resolving this typically involves reviewing your rewrite conditions and ensuring they include proper exit points.

Reading Nginx Error Logs

Nginx stores error logs at /var/log/nginx/error.log by default, and each virtual host can have its own error log configured within its server block. When you are managing multiple sites on one server, checking the appropriate virtual host log often points directly to the problem.

tail -f /var/log/nginx/error.log

Nginx error log entries have a similar structure to Apache's:

2026/05/20 09:45:23 [error] 2345#2345: *6789 open() "/var/www/html/missing-file.jpg" 
failed (2: No such file or directory), client: 192.168.1.60, server: example.com, 
request: "GET /images/missing-file.jpg HTTP/1.1", host: "example.com"

This shows a 404 error caused by a missing file. The open() failed (2: No such file or directory) pattern appears frequently in Nginx logs and usually means a file path is incorrect or the file was deleted without updating the configuration. If you are distributing traffic across multiple servers using load balancing, understanding these logs helps identify which backend is experiencing problems. There is a detailed explanation of how Nginx load balancing works if you want to understand the underlying mechanism.

Access Logs: What They Tell You

Access logs record every request the web server handles. While they are primarily used for analytics and traffic monitoring, they are also invaluable when debugging. If a specific endpoint is failing, the access log shows whether requests are reaching the server and what response code is being returned.

Apache Combined Log Format looks like this:

192.168.1.50 - - [20/May/2026:09:45:23 +0000] "GET /api/users HTTP/1.1" 200 1234 
"https://example.com/page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"

The fields are: client IP, ident (usually -), user (usually -), timestamp, request line, status code, response size, referrer, and user agent. The status code is the most important field for debugging. A 200 means success. 404 means not found. 500 means the server encountered an error while processing the request.

grep " 500 " /var/log/apache2/access.log | tail -20

This finds the 20 most recent server error responses, which are the ones most likely to indicate application problems. You can adapt this for Nginx by changing the log path to /var/log/nginx/access.log.

PHP Error Logs

PHP errors can go to different places depending on the configuration. The php.ini directives error_log and display_errors determine where and whether errors are shown. On production servers, display_errors should typically be Off to avoid exposing internal details to visitors.

error_log = /var/log/php_errors.log
display_errors = Off
log_errors = On

If PHP is running under Apache or Nginx, PHP errors often appear in the web server's error log. If PHP-FPM is used, errors may go to /var/log/php-fpm/ instead. Checking both locations ensures you do not miss relevant entries.

tail -f /var/log/php_errors.log

Common PHP errors and what they mean:

  • Fatal Error: Class 'PDO' not found: The PDO extension is not installed or enabled. Check php.ini or install the extension using your package manager.
  • Maximum execution time exceeded: A PHP script ran longer than the max_execution_time limit. Either optimise the script or increase the limit in php.ini.
  • Allowed memory size exhausted: The script exceeded the memory_limit. Either the script needs more memory or there is an infinite loop or memory leak in the code.
sudo apt install php-mysql
sudo systemctl restart php-fpm

These commands install the MySQL extension for PHP and restart the PHP-FPM service. Always test after making configuration changes to ensure the issue is resolved.

Reading the Syslog

The syslog records events from system services using a standard format: a timestamp, a hostname, a service name, and a message. It provides a centralised view of what the operating system and core services are doing.

May 20 09:45:23 server01 systemd: nginx.service: Main process exited, code=exited, status=1/FAILURE

This entry tells you that the Nginx service stopped unexpectedly. The status=1/FAILURE indicates the process exited with a failure code. Search syslog for service failures when a particular service is not running as expected.

grep -E "FAILURE|ERROR|CRITICAL" /var/log/syslog | tail -30

This finds the most recent error-level messages across all services. You can narrow this further by piping to grep with a specific service name if you know which service is causing problems.

Log Rotation: Managing Log File Size

Log files grow continuously. Without rotation, they eventually consume all available disk space, which can cause services to fail or the system to become unresponsive. Linux uses logrotate to manage this. Logrotate compresses old logs, creates new ones, and deletes logs older than a configurable retention period.

/etc/logrotate.d/nginx {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        /bin/kill -USR1 $(cat /run/nginx.pid 2>/dev/null) 2>/dev/null || true
    endscript
}

This configuration rotates Nginx logs daily, keeps 14 days of history, compresses old logs with gzip, and signals Nginx to reopen its log files after rotation. Understanding logrotate prevents confusion when looking for older log files that have been compressed into .gz archives. If you are troubleshooting an issue that happened several days ago, you may need to look inside those compressed files using zgrep or zcat.

Using Journalctl for Systemd Services

On systems using systemd, the journalctl command retrieves logs from the systemd journal, which is a structured binary log. It provides more filtering options than reading plain text log files and stores data in a more queryable format.

journalctl -u nginx.service --since "1 hour ago"

This shows Nginx logs from the last hour. Combine with -f to follow in real time, similar to tail -f on text files.

journalctl -u nginx.service -f

Filter by priority level to see only errors and above:

journalctl -p err -b

The -b flag limits output to the current boot. This is useful when you have just restarted a service and want to see only the relevant messages from this session. Use --since and --until with datetime strings for precise time ranges:

journalctl -u nginx.service --since "2026-05-20 09:00:00" --until "2026-05-20 10:00:00"

If you are running a mail server alongside your web server, you might find it useful to understand how services like Postfix handle their logging as well. There is a guide to Postfix configuration that covers mail server logging in more detail.

Real-World Debugging with Logs

A practical example: a PHP application reports that database connections are failing. Where do you look? Working through the layers systematically pinpoints the problem faster than guessing.

  1. Check the application error log for any PHP errors or exceptions that the application itself has recorded.
  2. Check /var/log/mysql/error.log for database-level errors, such as connection limits or authentication failures.
  3. Check /var/log/syslog for MySQL service restarts, out-of-memory kills, or other system-level events affecting the database.
  4. Check /var/log/apache2/error.log if Apache is proxying to PHP-FPM, as some errors appear there instead of in PHP's own log.

A missing MySQL error log entry when application connections fail often means the application is not reaching MySQL at all. In that case, check the MySQL bind address configuration and firewall rules to ensure the connection is not being blocked at the network level.

Another common scenario: a site loads slowly or returns 502 Bad Gateway. Nginx logs the error with the upstream response time. A 502 usually means the PHP-FPM worker processes crashed or the upstream connection timed out. If your Nginx server is distributing requests to multiple backends, checking the load balancer configuration can reveal whether certain servers are being overwhelmed or marked as unavailable.

grep -E "502|504|upstream timed out" /var/log/nginx/error.log

If PHP-FPM appears to be the culprit, check the PHP-FPM error log and the process manager status:

ps aux | grep php-fpm | head -10
sudo systemctl status php-fpm

Setting Up Basic Log Monitoring

On servers with many services, reading logs across multiple files is inefficient. Centralised logging tools aggregate logs from all servers into one interface where you can search and filter across everything at once. For smaller setups, a simple shell script can send critical log entries to a monitoring service or email address.

#!/bin/bash
# Alert if Apache error log has new entries since last check

LOG_FILE="/var/log/apache2/error.log"
STATE_FILE="/tmp/apache_error_watermark"

CURRENT_SIZE=$(stat -c%s "$LOG_FILE")

if [ -f "$STATE_FILE" ]; then
    LAST_SIZE=$(cat "$STATE_FILE")
    if [ "$CURRENT_SIZE" -gt "$LAST_SIZE" ]; then
        tail -c +$((LAST_SIZE + 1)) "$LOG_FILE" | grep -iE "error|critical|fatal" | \
        mail -s "Apache Errors on $(hostname)" [email protected]
    fi
fi

echo "$CURRENT_SIZE" > "$STATE_FILE"

Run this script every five minutes via cron. It detects new error entries since the last run and sends an email alert. This is a lightweight alternative to a full monitoring stack for a single server or small number of servers. For production environments with multiple servers, tools like the ELK stack or Grafana Loki provide more scalable solutions, but starting simple helps you understand what you actually need before adding complexity.

Common Log Patterns to Recognise

Certain log patterns appear frequently once you know what to look for. Recognising them speeds up diagnosis significantly.

  • Permission denied: Usually means the web server user (www-data or nginx) does not have read or write access to the requested file or directory.
  • Connection refused: The service is not listening on the expected port, or a firewall is blocking the connection.
  • Too many open files: The server has hit its ulimit for file descriptors. Increase the limit in /etc/security/limits.conf or the service configuration.
  • Out of memory: The OOM killer terminated a process. Check dmesg or /var/log/kern.log for OOM events.