Is Your Server Disk Almost Full? Safely Handle Growing Logs

A server that suddenly slows down, fails to run processes, or rejects uploads may not necessarily be experiencing a traffic spike. One often overlooked cause is a nearly full disk due to continuously growing log files.

Logs are records of application and system activities: who accessed the website, errors that occurred, processes that failed, and messages from certain services. These records are very useful during troubleshooting. The problem is, logs do not always stop growing. If not managed, a single file can reach gigabyte sizes and gradually consume all available storage space.

Simply deleting all files in the /var/log folder is not enough. Some logs may still be needed for investigations, audits, or ongoing application processes. A safer approach is to identify the source of the waste, retain the important parts, and then implement log rotation.

Start by Checking Disk Space

Do not guess which folder might be problematic. First, check the capacity of each partition using the command:

df -h

This command displays total capacity, used space, and remaining space in a human-readable format. Pay attention to the percentage usage column. Partitions nearing 90 percent should be checked immediately, as the server needs free space to create temporary files, write new logs, and run normal processes.

Once you identify the full partition, look for the largest folders. For an initial check, use:

sudo du -xhd1 /var | sort -h

The -d1 option limits the results to one folder level, while sort -h sorts the sizes to make the largest folders easier to spot. If /var/log turns out to be the largest, continue the check:

sudo du -xhd1 /var/log | sort -h

Commonly large files include syslog, auth.log, nginx, apache2, or logs from applications running with systemd.

Differentiate Between Active Logs and Archives

Before deleting files, distinguish between active logs and old archives. Active files are usually still open by services. Deleting them directly may free up space in some cases, but processes still holding those files can continue writing to the old inode. As a result, disk space may not be fully reclaimed until the service is restarted.

To check if a file is being used by a process, use:

sudo lsof +L1

This command can help find files that have been deleted but are still open. Such conditions often make the output of df -h appear full, while the output of du does not find files of equivalent size.

For logs that are no longer needed and are truly old archives, deletion can be done according to storage policies. However, do not make commands like rm -rf /var/log/* a standard solution. This method risks deleting folder structures, permissions, or logs still needed by services.

Use Log Rotation, Not Manual Deletion

Log rotation is a mechanism for periodically replacing active logs. Old files are usually renamed, compressed, and deleted after reaching a certain number or age.

In many Linux distributions, this setting is managed by logrotate. You can check the main configuration with:

cat /etc/logrotate.conf

Application-specific configurations are usually located at:

ls -lah /etc/logrotate.d/

Here’s a simple configuration example for application logs:

/var/log/aplikasi/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

This means logs are rotated daily and kept for up to 14 versions. Old files are compressed to save space. missingok prevents the process from failing if a file is not present, while notifempty avoids creating archives for empty logs.

The copytruncate option can be useful for applications that do not support reopening log files. However, there is a small delay when the file is copied and truncated, so it may not always be the best choice. If the application supports reloading or reopening logs, that mechanism is usually cleaner.

Check the Causes of Log Growth

Rotation only controls log growth. If files are growing very quickly, the underlying causes still need to be investigated.

• Repeated errors: the application may be failing to connect to a database or certain services on every request.
• Debug mode active: a production environment still using debug log levels can generate far more records.
• Suspicious bots or requests: the website may receive a large number of automated requests, especially to non-existent endpoints.
• Overly verbose configuration: some applications log payloads, queries, or connection details that are not needed all the time.
• Repeated service failures: processes that continuously crash and restart can write error messages endlessly.

Use part of the log content to look for patterns, not just delete it. For example:

sudo tail -n 100 /var/log/nginx/error.log

If you want to count the most frequently occurring messages, you can copy the logs to your working location first, then analyze them with grep, awk, or the observability tools used by your team.

If the Disk is Critical, Perform Gradual Recovery

When there is very little free space left, the priority is to create enough working space without disrupting main services.

1. Temporarily stop processes that are continuously generating errors, if it is safe to do so.
2. Compress old log archives that still need to be retained.
3. Move archives to separate storage according to security and retention policies.
4. Run log rotation manually using test mode first.
5. Check capacity again with df -h.
6. Once space is available, investigate the causes of errors before returning the service to normal operation.

Do not move logs to any public storage just to free up disk space. Logs may contain IP addresses, usernames, tokens that were accidentally recorded, or internal application information. Treat log archives like operational data, not trash files.

What You Can Do Now

If you manage a VPS or website server, set up a simple routine check: monitor disk usage, review the largest log folders, ensure rotation is functioning, and establish a reasonable retention period. For small servers, keeping active logs for 7 to 14 days may be sufficient. For audit or investigation needs, the policy may differ.

Importantly, these decisions should be made before problems arise. Logs should help answer the question, “what happened?”, not become the reason the server stops working. With proper rotation, capacity monitoring, and habits of reading error patterns, you can save space without losing important traces during troubleshooting.

– Rio Yotto @rioyotto