@nateberkopec @onyxraven @getajobmike Scaling is not always a cloud thing, but sometimes a per-host thing. If the server is idle, it doesn't make sense to spawn n/2 processes when you have 32 vCores. It's a waste of resources.
If the queue size is increasing, then it might be a good idea to spawn new sidekiqs and bring the queue down - until it is down to a reasonable size.

Yes, that's more complex than just spawn n/2 processes and you'll need some kind of monitoring the sidekiq queue length. Maybe there is already something available.

For example, if you spawn a new sidekiq for every 2 or 5 minutes of backlog (until a upper limit of n processes is reached), then it would autoscale based on the queue length.
No need to invoke some cloud magic or provider or additional hardware.

1
Share
Share on Mastodon
Share on Twitter
Share on Facebook
Share on Linkedin
nate berkopec

@ij How is it a waste of resources? ... the cores are sitting there just idling otherwise?

1
3mo
Replies