[Awesome Ruby Gem] Use sidekiq-unique-jobs gem to adds unique constraints to sidekiq jobs

Posted on 2021-07-03 Edited on 2023-09-02 In Programming Language , Ruby , Awesome Ruby Gem Views: Word count in article: 4.5k Reading time ≈ 4 mins.

sidekiq-unique-jobs

sidekiq-unique-jobs adds unique constraints to sidekiq jobs. The uniqueness is achieved by creating a set of keys in redis based off of queue, class, args (in the sidekiq job hash).

Installation

You can install it as a gem:

1	$ gem install sidekiq-unique-jobs

or add it into a Gemfile (Bundler):

# Gemfile

# sidekiq-unique-jobs | RubyGems.org | your community gem host
# https://rubygems.org/gems/sidekiq-unique-jobs
gem 'sidekiq-unique-jobs', `7.1.2'

Configuration

Sidekiq.configure_server do |config|
  config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }

  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end

  config.server_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Server
  end

  SidekiqUniqueJobs::Server.configure(config)
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }

  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end
end

Usages

Your first worker

The most likely to be used worker is :until_executed. This type of lock creates a lock from when UntilExecutedWorker.perform_async is called until right after UntilExecutedWorker.new.perform has been called.

# frozen_string_literal: true

class UntilExecutedWorker
  include Sidekiq::Worker

  sidekiq_options queue: :until_executed,
                  lock: :until_executed

  # sidekiq_options lock: :until_executed
  # sidekiq_options lock: :until_expired, lock_ttl: 1.day
  # sidekiq_options lock: :until_and_while_executing,
  #                 lock_timeout: 2
  # sidekiq_options lock: :while_executing,
  #                 lock_timeout: 2,

  def perform
    logger.info("cowboy")
    sleep(1) # hardcore processing
    logger.info("beebop")
  end
end

NOTE Unless a conflict strategy of :raise is specified, if lock fails, the job will be dropped without notice. When told to raise, the job will be put back and retried. It would also be possible to use :reschedule with this lock.

NOTE Unless this job is configured with a lock_timeout: nil or lock_timeout: > 0 then all jobs that are attempted to be executed will just be dropped without waiting.

There is an example of this to try it out in the myapp application. Run foreman start in the root of the directory and open the url: localhost:5000/work/duplicate_while_executing.

In the console you should see something like:

0:32:24 worker.1 | 2017-04-23T08:32:24.955Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: start
10:32:24 worker.1 | 2017-04-23T08:32:24.956Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: start
10:32:24 worker.1 | 2017-04-23T08:32:24.957Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: start
10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: start
10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 WhileExecutingWorker INFO: perform(1, 2)
10:32:34 worker.1 | 2017-04-23T08:32:34.964Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: done: 10.009 sec
10:32:34 worker.1 | 2017-04-23T08:32:34.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 WhileExecutingWorker INFO: perform(1, 2)
10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 WhileExecutingWorker INFO: perform(1, 2)
10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: done: 20.009 sec
10:32:54 worker.1 | 2017-04-23T08:32:54.970Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f WhileExecutingWorker INFO: perform(1, 2)
10:32:54 worker.1 | 2017-04-23T08:32:54.969Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: done: 30.012 sec
10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec