When the number of visitors increases, a common solution is to enable autoscaling. Simply put, autoscaling allows the cloud to add or remove servers based on workload. This idea makes sense: when traffic is high, capacity increases; when it's low, resources are reduced to avoid ongoing costs.
The problem is, autoscaling is not a magic button. If the application stores sessions on a single server, the database has become a bottleneck, or the traffic spike ends before the new server is ready, adding instances can complicate the system and increase bills. Autoscaling works well when the application's foundation, measurements, and spending limits have been considered.
What exactly does autoscaling do?
There are two common approaches. Scale up means increasing the capacity of a single machine, for example, by upgrading a VPS with larger CPU and RAM specifications. Scale out means adding multiple instances so that the load is distributed across many machines.
Autoscaling is usually closer to scale out. The cloud monitors metrics such as CPU usage, number of requests, queue length, or application-specific metrics. If certain conditions are met, the system adds instances. When the load decreases, some instances can be shut down.
Cloud architecture documentation generally emphasizes that scale out requires applications capable of running on more than one instance. This means user requests should not depend on a specific server. Applications should also be stateless, meaning session information and important status are stored in shared services, not just in the memory of a single machine.
Why is CPU often a poor signal?
CPU is easy to understand and available in almost all cloud services. However, CPU is not always the main cause of application slowness.
- Catalog applications can experience database queues even when CPU is low.
- Image processing workers may be better scaled based on queue length.
- Payment APIs should be monitored through latency, error rate, and the number of requests being processed.
- Node.js applications may appear underutilized on several cores but still experience event loop congestion.
Therefore, autoscaling metrics should be related to how the application operates. For web services, requests per second or p95 latency—the response time that includes 95 percent of the fastest requests—are sometimes more relevant than CPU. For queue-based systems, the number of unprocessed messages can be a more sensible indicator.
Analysis: the best metrics are not the easiest to find on the dashboard, but those closest to user experience or business capacity. If customers are waiting too long due to order queues, a CPU of 45 percent does not mean the system is fine.
Autoscaling does not happen instantly
One common misconception is to assume that new servers are immediately available once metrics exceed a threshold. In reality, the system needs to detect conditions, make decisions, create instances, run boot processes, configure settings, and then add those instances behind the load balancer.
This process can take time. If a traffic spike lasts only one or two minutes, additional servers may only be ready when the load has already decreased. For predictable traffic patterns, scheduled scaling makes more sense. For example, increasing capacity 15 minutes before a promotional period starts.
For unpredictable spikes, reactive autoscaling is still useful, but it needs to be combined with caching, CDN, queues, request throttling, or graceful degradation mechanisms. Non-critical features, such as product recommendations or real-time reports, can be temporarily delayed to keep core functions running.
Deliberately set minimum and maximum limits
Minimum and maximum configurations are not just technical numbers. The minimum value determines the baseline capacity that is always available, while the maximum serves as a fence to prevent the system from adding resources indefinitely.
A minimum that is too low can lead to cold starts, long queues, or loss of redundant capacity. Conversely, a minimum that is too high keeps costs high even when traffic is low.
The maximum limit also needs to be adjusted according to the capabilities of other components. If an application can open up to 50 instances, but the database can only handle 10 active connections per instance, autoscaling compute can accelerate damage at the database layer.
Adding application servers is not the same as increasing the capacity of the entire system.
Start with conservative numbers. Monitor resource usage, response times, error rates, and costs over several weeks. After that, gradually increase the limits based on real data, not optimistic estimates.
Beware of flapping and rising bills
Flapping occurs when the system scales out and scales in too frequently because the thresholds are too close together. For example, an application adds instances when CPU exceeds 70 percent, then immediately removes them when it drops to 65 percent. If the load fluctuates around those numbers, the number of instances can rise and fall repeatedly.
Use sufficient distance between scale out and scale in rules. Also, allow time for new instances to truly receive traffic before the next decision is made. Some platforms provide cooldowns or scale-in rate controls to prevent overly aggressive capacity reductions.
Don't just send notifications when autoscaling succeeds. Also log when metrics are unavailable, when maximum limits are reached, and when scaling actions fail. Without these logs, the team might think the system is elastic while the application continues to operate at minimum capacity.
Checklist before enabling autoscaling
- Ensure the application can run on multiple instances. Avoid storing user sessions, temporary upload files, or important states only on the server's local disk.
- Define metrics based on real bottlenecks. Compare CPU, memory, latency, requests per second, database connections, and queue lengths.
- Test scale out and scale in. Ensure new instances receive the correct configuration and that deleted instances do not interrupt active work.
- Set safe minimum, maximum, and default values. These limits should consider the capacity of the database, cache, queue, and budget.
- Prepare cost protection. Use budget alerts, notifications when maximum limits are reached, and review bills after testing.
- Have a plan when maximum capacity is reached. Throttling, queuing, or disabling non-critical features is better than allowing all requests to fail simultaneously.
What does this mean for us?
For blogs, company profiles, or small applications with stable traffic, autoscaling may not yet be a priority. Caching, query optimization, CDN, and a single healthy server configuration often yield greater results with lower complexity.
Autoscaling becomes interesting when load patterns change, the cost of backup capacity feels heavy, or applications need to continue serving users during traffic increases. However, its benefits are only felt if the system is already measurable and its components do not depend on a single machine.
The safest step is not to immediately raise the instance limits, but to conduct small load tests. Measure when latency begins to worsen, which components become bottlenecks, how long new instances take to be ready, and what additional costs arise. From there, autoscaling becomes a data-driven decision—not just a feature checked off because it's available on the cloud dashboard.
Sources & further reading
- Autoscaling Guidance - Azure Architecture Center
- Best practices for autoscale - Azure Monitor
- Autoscaling groups of instances - Google Cloud Documentation
- Scaling cooldowns for Amazon EC2 Auto Scaling
– Rio Yotto @rioyotto