2 minutes
Why We Built a Custom Kubernetes Autoscaler, and the Math Behind It
I recently published an article on the Hydrolix blog about hdx-scaler, the custom Kubernetes autoscaler we built and now run on 200+ production clusters.
Read the full article: Why We Built A Custom Kubernetes Autoscaler, and the Math Behind It
Overview
In the article, I cover:
The problem: Our intake services were rejecting customer traffic with 429 errors, and on clusters without a request cap the same overload ended in OOM kills. CPU sat at a healthy-looking 35%, so the built-in autoscaler saw no reason to add pods.
Why standard autoscaling couldn’t save us: The built-in HPA scales on CPU and memory, which weren’t the failure signals. KEDA needs a cluster-wide metrics adapter that we can’t take over in customer-owned clusters. The standard VPA reacts to OOMs slowly and applies changes by evicting pods.
What we built: A single Go daemon that scrapes pods directly, treats OOM kills and evictions as events, handles both horizontal and vertical scaling, and makes decisions every 5 seconds on any metric we choose.
The math behind it: EWMA smoothing with bias correction, reservoir sampling, deadbands and cooldowns on the horizontal side; high-water marks, headroom calculations and a prioritized cascade of strategies on the vertical side.
Production results: Over 30 days, roughly 2 million scale-ups and 2 million scale-downs across 200+ clusters, and 37% less reserved memory on one worker fleet.
If you’re hitting the limits of CPU-based autoscaling or wondering what it takes to build your own, I hope you find this article useful.
250 Words
2026-11-10