Auto-Scaling

Auto-scaling policies define when and how to scale based on metrics like CPU utilization, request queue depth, memory usage, or custom application metrics. Scaling can be reactive, responding to current conditions, or predictive, anticipating demand based on historical patterns. Effective auto-scaling requires correctly identifying the bottleneck metric and setting appropriate thresholds and cooldown periods to prevent oscillation.

For AI product teams, auto-scaling is essential but challenging because AI inference workloads have unique scaling characteristics. GPU instances take longer to provision than CPU instances, model loading adds startup time, and inference latency may not correlate linearly with CPU or memory utilization. Teams often use custom metrics like inference queue depth or p99 latency as scaling triggers. Growth teams should understand auto-scaling behavior because growth campaigns, product launches, and viral moments can generate sudden traffic spikes that test scaling limits. Pre-warming inference capacity before predictable traffic events like marketing campaigns prevents degraded AI feature performance during the moments when user first impressions matter most.