Is Your Disk Server Filling Up Quickly? It Might Be Application Logs That Never Stop

A server that suddenly runs out of disk space often leads people to think about databases, file uploads, or backups. However, the cause can be simpler: application logs that keep growing without ever being rotated or deleted.

Logs are records of system activity—such as HTTP requests, application errors, login processes, or messages from containers. They are very useful when we need to find the cause of disruptions. However, logs are not eternal archives. If every request is logged too detailed and stored indefinitely, a small server can run out of space in just a few days.

Why can logs become a big problem?

Think of logs like an operational notebook. One entry may be small, but if an application receives thousands of requests every hour, the amount can quickly accumulate. Messages like GET /api/products, status codes, IP addresses, and access times may only take up a few hundred bytes. At scale, their accumulation can reach gigabytes.

The problem is not just about storage. When the disk is nearly full, applications may fail to write temporary files, databases may struggle to create additional files, deployment processes may halt, and even the operating system can behave unstably. Therefore, logs should be treated as part of operational design, not just as additional output from applications.

Docker itself warns that the json-file driver can cause log files to grow if rotation is not configured. Docker documentation recommends using log rotation or the local driver to reduce the risk of a full disk. ([docs.docker.com](https://docs.docker.com/engine/logging/configure/?utm_source=openai))

Start by finding out who is consuming the disk

Do not randomly delete log folders. The first step is to find the locations that are using the most space.

df -h
sudo du -sh /var/log/* 2>/dev/null | sort -h
sudo du -sh /var/lib/docker/* 2>/dev/null | sort -h

The df -h command provides an overview of partition usage. Meanwhile, du helps identify which folders are the largest. On servers running Docker, pay attention to the Docker data directory as container logs are usually stored in that area.

If disk usage is approaching 90 percent, conduct a more targeted check. Look for large files that have changed recently, check specific application logs, and see if there are processes generating repeated errors. Logs that continuously record stack traces or connection failure messages are usually the main suspects.

Differentiate between active logs, old logs, and important data

Not all log files are safe to delete. There are logs that are still being written by applications, old logs that have been rotated, and audit logs that must be retained longer due to business needs or compliance.

Avoid commands like deleting everything in /var/log without understanding the impact. Files that are being used by active processes can cause confusion when services try to rewrite them. For systemd journal, for example, there is a specific mechanism like journalctl --vacuum-time=7d to delete journal archives older than seven days. The systemd documentation also provides options based on size and number of files. ([freedesktop.org](https://www.freedesktop.org/software/systemd/man/255/journalctl.html?utm_source=openai))

Here’s an example of a more cautious cleanup:

sudo journalctl --disk-usage
sudo journalctl --vacuum-time=7d

The seven-day figure is not a universal rule. Choose retention periods based on troubleshooting needs. For simple applications, a few days may be sufficient. For financial systems or services that require incident investigation, certain logs may need to be sent to separate storage and retained longer.

Implement log rotation, rather than relying on manual cleanup

Manually cleaning logs only solves today’s problem. A healthier solution is log rotation, which is a mechanism that replaces active log files when they reach a certain size or age, then compresses or deletes old files.

For Docker containers, one example configuration is:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

With this configuration, each container is limited to a few log files of a certain size. Docker documentation states that the local driver uses automatic rotation, with default values limiting log size per container. Keep in mind that changes to daemon configuration usually apply to containers created after the configuration is applied; older containers need to be checked and reconfigured if necessary. ([docs.docker.com](https://docs.docker.com/engine/logging/drivers/local/?utm_source=openai))

If using Nginx, Apache, Node.js applications, or other Linux services that write files directly to disk, check the logrotate configuration. Ensure that old files are compressed, the number of copies is limited, and the application is notified to open new files after rotation.

Send important logs to a separate location

Local rotation is important, but do not make it the only copy. If the server fails, local logs can be lost as well. For logs that are important for monitoring and investigation, use a centralized service or send them to a separate log server.

For example, CloudWatch Logs can store logs from instances, applications, and other services. However, retention needs to be configured because by default logs can be stored indefinitely. AWS provides retention settings per log group, allowing teams to choose appropriate retention periods. ([docs.aws.amazon.com](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html?utm_source=openai))

A sensible strategy is to keep operational logs that are frequently queried for a short period, such as 7–30 days, and then archive audit or security logs to cheaper storage. Do not treat all logs the same. Highly detailed debug logs do not need to be treated the same as transaction records or administrative activity.

What you can do now

1. Check disk capacity: run df -h and look for partitions that are nearly full.
2. Identify the largest sources: check /var/log, Docker data, cache, file uploads, and local backups.
3. Stop the source of log explosions: look for repeated errors or overly detailed logging levels.
4. Enable rotation: set size and file limits for each service or container.
5. Determine retention periods: separate logs for debugging, security, audit, and business needs.
6. Use centralized storage: send important logs off the main server and limit access.
7. Create alarms: monitor disk usage, for example, when crossing 70, 80, and 90 percent.

The Bottom Line

A server full of logs is not a difficult problem if caught early. What is dangerous is leaving it unchecked, then acting only when the application has failed to write data.

Good logs are not logs that store everything forever. Good logs are those that are detailed enough to help make decisions, have clear retention periods, are protected from arbitrary deletion, and are not allowed to consume server workspace.

Sources & Further Reading

• Docker Docs: Configure logging drivers
• Docker Docs: Local file logging driver
• systemd journalctl documentation
• AWS CloudWatch Logs: Working with log groups and log streams
• AWS CloudWatch Logs overview

– Rio Yotto @rioyotto