Rails — Shared Redis connection pool between Sidekiq and Application

Connection Pooling is a cache of database connections that can be reused when future requests to the database are required. So, does that mean you open all the connections on the application boot up? No, you would rather just define the minimum pool of connection required and then lazy load them.

Just in case you’re not familiar with Sidekiq read my previous article.

We will connection pool and use it both for the application requirements and background jobs processors like Sidekiq, Resque, ActiveJob, etc.

Install Connection Pool gem (not required if you already have sidekiq installed.)

gem install connection_pool

Now in your sidekiq initializer define the Redis configuration as:

# frozen_string_literal: true
# using pool to share connection b/w application and sidekiq
# sidekiq required minimum connection of (2 + concurrency)
# https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/redis_connection.rb#L51
REDIS_POOL = ConnectionPool.new(size: 10) { Redis.new(url: ENV['REDIS_URL']) }
Sidekiq.configure_server do |config|
  config.redis = REDIS_POOL
end
Sidekiq.configure_client do |config|
  config.redis = REDIS_POOL
end

Note: Sidekiq needs 2 connections by default for a bunch of stuff like a heartbeat, maintain sentinel, etc. Read here. That’s why if you have max threads or concurrency of for ex. 5 then you need to have at least 7 connections in your pool.

The sidekiq workers will utilize the Redis pool for processing background jobs. To use the pool in the application —

# store `foo` with value `bar` in redis
REDIS_POOL.with do |client|
  client.set('foo', 'bar')
end
# retrieve value of key `foo`
REDIS_POOL.with do |client|
  client.get('foo')
end

All Redis operations can be performed inside the block. Brush up on Redis here.

Benefits of using Connection Pool:

1. Connections are lazily created. No unnecessary connections created until required.
2. Ruby Garbage Collection takes care of orphaned connections and cleans them.

Though as per the documentation of connection pool your connections should be self-repairing as there is no provision for health checking.

More on Sidekiq and Rails:

1. Configuring Rails 5 with Sidekiq Adapter and Redis Queues.
2. Configuring Sidekiq to use Cron Scheduler.