Traffic surges often occur at inconvenient times: a new promotion starts, an article suddenly gets shared widely, or an application is being used simultaneously by the entire team. In such situations, a single server with fixed capacity can become a weak point. The CPU gets maxed out, response times slow down, and users start encountering errors.
Autoscaling is designed to handle such situations. Simply put, autoscaling is a mechanism that adds or reduces computing capacity based on load. However, this feature is not a magic button. If the application is not ready to run on more than one instance, the database becomes a bottleneck, or the scaling rules are incorrect, autoscaling will only increase costs without making the service truly more stable.
Autoscaling means adjusting capacity, not just adding servers
There are two common approaches to increasing application capacity. Vertical scaling means upgrading the specifications of the same machine, such as adding CPU or RAM. This method is relatively easy but is still limited by the size of available machines and often requires a restart process.
Horizontal scaling means adding the number of instances running the application. For example, an application that usually runs on two servers can be expanded to five servers when traffic increases, then reduced back when the load decreases. Cloud services generally use this approach because it is more flexible in handling traffic changes.
Azure documentation explains that autoscale can use metrics such as CPU usage, queue length, and memory as a basis for decision-making. In Kubernetes, the Horizontal Pod Autoscaler mechanism can adjust the number of pods based on observed metrics. This means that autoscaling is not guesswork; it operates from pre-configured signals.
Why CPU alone is often not a sufficient indicator?
Rules like “add servers when CPU exceeds 70 percent” are indeed easy to understand. However, CPU is not always the main cause of slow applications. An online store website may experience long queues because the database runs out of connections. A payment API may lag due to slow third-party services. An image processing application might be waiting on disk or storage, not the processor.
Therefore, scaling metrics need to be adjusted according to how the application works. Some more relevant examples include:
- Requests per second for API services that receive many short requests.
- Latency for applications that need to respond quickly.
- Queue length for systems that process jobs incrementally.
- Memory usage for applications that store a lot of data in memory.
- Database connection count when the database is the busiest part of the system.
For queue-based jobs, the number of waiting messages is often more useful than CPU. If the queue keeps growing, adding workers may be more appropriate than adding web instances.
Parts of the application that must be ready before scaling out
Scaling out only works well if each instance can serve requests relatively independently. One common issue is storing user sessions in the server's local memory. When a user moves from Server A to Server B, the session cannot be found, and the user may suddenly be considered logged out.
The solution can be centralized session storage, such as Redis or a database, or using authentication mechanisms that do not rely on local memory. File uploads should also not be stored only on the local disk of the instance. When an instance is replaced or deleted, those files can be lost if not stored in object storage or volumes designed for sharing.
Also, pay attention to the application startup process. New instances are not immediately ready to receive traffic. The application may still be running migrations, loading configurations, or connecting to other services. Use health checks and readiness checks so that the load balancer only sends requests to instances that are truly ready.
Autoscaling does not solve database bottlenecks
Adding ten application servers does not help much if they are all waiting on the same database. In fact, overly aggressive scaling out can worsen the situation because the number of connections to the database increases rapidly.
Before enabling autoscaling, check the following:
- Do frequently used queries have appropriate indexes?
- Does the connection pool have a reasonable limit?
- Can frequently read data be assisted with caching?
- Can heavy jobs be moved to background workers?
- Does the database have clear capacity limits and upgrade plans?
In many cases, job queues, caching, and query optimizations yield greater results than simply adding application servers.
Set minimum, maximum, and change rate limits
Autoscaling requires clear limits. The minimum value ensures that the application maintains a basic capacity, while the maximum value prevents the system from continuously adding resources during anomalies or traffic attacks.
The rate of change is also important. If the system immediately adds many instances just because the CPU spikes briefly, capacity can swing up and down. This pattern is often referred to as thrashing: the system reacts too quickly to small changes.
Use reasonable evaluation periods and wait times between scaling out and scaling in. Scaling in usually needs to be slower than scaling out. Quickly adding capacity can help maintain service, while removing capacity too quickly risks overwhelming the application again.
What can be done now
- Record traffic patterns. Observe peak hours, request counts, latency, error rates, and resource usage over several days.
- Define key metrics. Choose metrics that are closest to user impact, not just the easiest metrics to observe.
- Test the application with two or more instances. Ensure sessions, uploads, caching, and configurations do not depend on specific servers.
- Set resource limits. Establish minimum and maximum instance values to keep costs predictable.
- Simulate traffic surges. Run controlled load tests and monitor the database, queue, storage, and external services.
- Prepare cost and performance alarms. Healthy autoscaling should be monitored from two sides: whether users receive better service and whether spending remains reasonable.
Good autoscaling is not a system that always adds servers. It is a system that knows when to increase capacity, when to stop, and when the real problems lie in other components.
What does this mean for us?
For small applications, autoscaling may not need to be implemented from day one. A single scalable VPS, good backups, monitoring, and recovery procedures are often sufficient for the initial stages. However, when traffic becomes unpredictable or downtime has a significant business impact, autoscaling can become an important part of infrastructure design.
Start with the application architecture, not the cloud configuration menu. Ensure the application can run on multiple instances, understand the main bottlenecks, and then create scaling rules based on real data. With this approach, autoscaling becomes a tool for maintaining reliability, not just an expensive feature that looks sophisticated.
For technical details, see the Azure autoscale documentation, AWS Auto Scaling documentation, and Kubernetes Horizontal Pod Autoscaler guide.
Sources & further reading
– Rio Yotto @rioyotto