Sidekiq Rundown Part 2: Optimizing Job Execution within Sidekiq
In the previous post of this series, we set up a simple Sidekiq worker. This time we introduce Sidekiq Batches and ways to configure Sidekiq Worker Concurrency.
While optimizing the performance of our applications and systems, we should ask ourselves, "where is the bottleneck?" What piece of code or infrastructure is impeding the flow of computing? Some bare metal solutions involve increasing a machine's resources in terms of CPU, RAM, I/O, or disk storage. Some less apparent solutions could involve some random configuration on a load balancer or an API gateway/proxy. Today we are focused on bottlenecks from within Sidekiq and how to optimize your sidekiq configuration. The goal is to perform heavy workloads efficiently and concurrently in the shortest execution time possible.
Sidekiq Batches
You can think of a batch as one big job or a container that knows about all of its jobs. One reason to use a batch is to group similar jobs to track the overall progress and completion of an encompassing task with statuses.
You can register callback methods on batches that fire after the batch of jobs has completed.
There are two scenarios where we can fire a callback.
on_success # Callback executes when a batch's jobs have 100% completed successfully.
on_completion # Callback executes when a batch's jobs have completed, which may include failed jobs.
These callbacks can be useful for chaining together batches in sequence or sending a completion notification for an important task.
Sidekiq Concurrency
When you start Sidekiq within your application it allocates a certain amount of threads from your laptop or server. By default, Sidekiq allocates 10 threads per worker which works fine for most use cases. If your job traffic starts to increase and your queue lengths keep growing, increase the concurrency of that worker if you have the resources available. Sidekiq experiences stability issues with a concurrency of 100 threads. The author of Sidekiq, Mike Perham, suggests a concurrency of no higher than 50. You want to make sure that your concurrency is low enough that your server is not close to maxing out it's I/O resources.
Using batches and concurrency together, we can quickly process jobs with better visibility and control. Finding the right balance in concurrency and resources can take some experimenting. Some good starting points are making sure that the number of connections in your database connection pool is greater than or equal to the number of threads available to all sidekiq workers.
First, let's take a look at creating a batch. Let's imagine that we have thousands of blog posts to proofread & edit and that we have written a BlogPostEditorWorker to process a blog post by blog_post_id.
class BlogPostEditorWorker
include Sidekiq::Worker
def perform(blog_post_id)
post = BlogPostClient.get(blog_post_id)
Editor.process_blog_post(post)
end
end
batch = Sidekiq::Batch.new
batch.description = "Processes Blog Posts"
batch.on(:success, EmailPublisherWithPosts, :to => Publisher.email_recipient
# Any worker.perform created inside the block below will belong to the batch
batch.jobs do
BlogPostClient.find(processed: false).each do |post|
BlogPostEditorWorker.perform_async(post.blog_post_id)
end
end
Common Problem Scenarios while Scaling Sidekiq Jobs
Scenario: You are running long Sidekiq jobs that consume a worker's threads, causing other jobs to move slowly through a congested queue.
Solution: Refactor your Sidekiq worker to process single atomic pieces of work instead of a long-running batch. Write worker code that runs smaller jobs that handle one individual unit of work at a time. This enables Sidekiq to leverage it's available concurrency as opposed to only using one thread.
Try to stay away from doing this:
batch.jobs do
one_thousand_blog_post_ids = BlogPostClient.find(processed: false).map{ |post| post.blog_post_id }
BlogPostEditorWorker.perform_async(one_thousand_blog_post_ids)
end
Prefer processing single items:
batch.jobs do
BlogPostClient.find(processed: false).each do |post|
BlogPostEditorWorker.perform_async(post.blog_post_id)
end
end
Scenario: You are putting thousands of asynchronous jobs into a queue which takes way too long. In the above example, we have to enqueue 1000 jobs. Doing this inside of .each is expensive as each new job is a trip to Redis
Solution: Enqueue several jobs at a time with a bulk_jobs call to the Sidekiq client. The main advantage here is that we are vastly reducing the number of network trips to Redis.
class BulkBlogPostEditorWorkerLoader
include Sidekiq::Worker
def perform(batch_id)
blog_posts = BlogPostClient.find(processed: false)
args = blog_posts.map{ |bp| [bp.blog_post_id] }
# args must be an array of arrays where each array
# holds parameters passed to the worker's perform method.
# puts args => [ ['123'], ['124'], ['125'] ... ]
Sidekiq::Client.push_bulk('class' => BlogPostEditorWorker, 'args' => args)
end
end
Scenario: You have a long-running job consuming a worker for 30 minutes and counting, blocking more critical jobs.
Solution: Add a worker with a reserved queue for critical jobs. Some services may rely on Sidekiq jobs to execute in near real-time. These jobs are better off working off of reserved queues. We can also give our critical queue 10 extra threads knowing we have a higher volume of critical items that need processing.
When starting your Sidekiq workers
bundle exec sidekiq -q default
bundle exec sidekiq -q critical -c 20
These were just a few tricks in Sidekiq to optimize your task workflows. Knowing the limitations of your downstream resources is vital when trying to scale a piece of your system. Also remember to experiment and keep track of execution times to find the optimal configuration for your system.
Sidekiq Resources
About the author
Logan Beougher is a Software Engineer at Custom Ink and has been an Inker since May 2019.
Interested in writing really fast code? We’re hiring! Visit us at customink.com/jobs
