Job queue

  Mojo::Server::Prefork                              +--------------+                     Minion::Worker
  |- Mojo::Server::Daemon [1]       enqueue job ->   |              |   -> dequeue job    |- Minion::Job [1]
  |- Mojo::Server::Daemon [2]                        |  PostgreSQL  |                     |- Minion::Job [2]
  |- Mojo::Server::Daemon [3]   retrieve result <-   |              |   <- store result   |- Minion::Job [3]
  +- Mojo::Server::Daemon [4]                        +--------------+                     |- Minion::Job [4]
                                                                                          +- Minion::Job [5]

They are not to be confused with time based job schedulers, such as cron or systemd timers. Both serve very different purposes, and cron jobs are in fact commonly used to enqueue Minion jobs that need to follow a schedule. For example to perform regular maintenance tasks.

Mojolicious

  use Mojolicious::Lite -signatures;

  plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'};

  # Slow task
  app->minion->add_task(poke_mojo => sub ($job, @args) {
    $job->app->ua->get('mojolicious.org');
    $job->app->log->debug('We have poked mojolicious.org for a visitor');
  });

  # Perform job in a background worker process
  get '/' => sub ($c) {
    $c->minion->enqueue('poke_mojo');
    $c->render(text => 'We will poke mojolicious.org for you soon.');
  };

  app->start;

Background worker processes are usually started with the command Minion::Command::minion::worker, which becomes automatically available when an application loads Mojolicious::Plugin::Minion.

  $ ./myapp.pl minion worker

The worker process will fork a new process for every job that is being processed. This allows for resources such as memory to be returned to the operating system once a job is finished. Perl fork is very fast, so don't worry about the overhead.

  Minion::Worker
  |- Minion::Job [1]
  |- Minion::Job [2]
  +- ...

By default up to four jobs will be processed in parallel, but that can be changed with configuration options or on demand with signals.

  $ ./myapp.pl minion worker -j 12

Jobs can be managed right from the command line with Minion::Command::minion::job.

  $ ./myapp.pl minion job

You can also add an admin ui to your application by loading the plugin Mojolicious::Plugin::Minion::Admin. Just make sure to secure access before making your application publicly accessible.

  # Make admin ui available under "/minion"
  plugin 'Minion::Admin';

Deployment

  [Unit]
  Description=My Mojolicious application workers
  After=postgresql.service

  [Service]
  Type=simple
  ExecStart=/home/sri/myapp/myapp.pl minion worker -m production
  KillMode=process

  [Install]
  WantedBy=multi-user.target

Consistency

                                                      +----------+
                                                      |          |
                                             +----->  | finished |
   +----------+            +--------+        |        |          |
   |          |            |        |        |        +----------+
   | inactive |  ------->  | active |  ------+
   |          |            |        |        |        +----------+
   +----------+            +--------+        |        |          |
                                             +----->  |  failed  |  -----+
        ^                                             |          |       |
        |                                             +----------+       |
        |                                                                |
        +----------------------------------------------------------------+

The system is eventually consistent and will preserve job results for as long as you like, depending on "remove_after" in Minion. But be aware that "failed" results are preserved indefinitely, and need to be manually removed by an administrator if they are out of automatic retries.

While individual workers can fail in the middle of processing a job, the system will detect this and ensure that no job is left in an uncertain state, depending on "missing_after" in Minion. Jobs that do not get processed after a certain amount of time, depending on "stuck_after" in Minion, will be considered stuck and fail automatically. So an admin can take a look and resolve the issue.

Priorities

  # Default priority
  $minion->enqueue('check_links', ['https://mojolicious.org']);

  # High priority
  $minion->enqueue('check_links', ['https://mojolicious.org'], {priority => 30});

  # Low priority
  $minion->enqueue('check_links', ['https://mojolicious.org'], {priority => -30});

You can use "retry" in Minion::Job to raise or lower the priority of a job.

  $job->retry({priority => 50});

Job results

  # Check job state
  my $state = $minion->job($job_id)->info->{state};

  # Get job result
  my $result = $minion->job($job_id)->info->{result};

While the "state" will be assigned automatically by Minion, the "result" for "finished" jobs is usually assigned manually with "finish" in Minion::Job.

  $minion->add_task(job_with_result => sub ($job) {
    sleep 5;
    $job->finish({message => 'This job should have taken about 5 seconds'});
  });

For jobs that "failed" due to an exception, that exception will be assigned as "result".

  $minion->add_task(job_that_fails => sub ($job) {
    sleep 5;
    die 'This job should always fail after 5 seconds';
  });

But jobs can also fail manually with "fail" in Minion::Job.

  $minion->add_task(job_that_fails_with_result => sub ($job) {
    sleep 5;
    $job->fail({errors => ['This job should fail after 5 seconds']});
  });

Retrieving job results is of course completely optional, and it is very common to have jobs where the result is unimportant.

Named queues

  # Use "default" queue
  $minion->enqueue('check_links', ['https://mojolicious.org']);

  # Use custom "important" queue
  $minion->enqueue('check_links', ['https://mojolicious.org'], {queue => 'important'});

For every named queue you can start as many workers as you like with the command Minion::Command::minion::worker. And each worker can process jobs from multiple named queues. So your workers can have overlapping responsibilities.

  $ ./myapp.pl minion worker -q default -q important

There is one special named queue called "minion_foreground" that you should avoid using directly. It is reserved for debugging jobs with "foreground" in Minion.

Job progress

  $minion->add_task(job_with_progress => sub ($job) {
    sleep 1;
    $job->note(progress => '25%');
    sleep 1;
    $job->note(progress => '50%');
    sleep 1;
    $job->note(progress => '75%');
    sleep 1;
    $job->note(progress => '100%');
  });

Notes, similar to job results, can be retrieved with "job" in Minion, all you need is the job id.

  # Get job metadata
  my $progress = $minion->job($job_id)->info->{notes}{progress};

You can also use notes to store arbitrary metadata with new jobs when you create them with "enqueue" in Minion.

  # Create job with metadata
  $minion->enqueue('job_with_progress', [], {notes => {progress => 0, something_else => [1, 2, 3]}});

The admin ui provided by Mojolicious::Plugin::Minion::Admin allows searching for jobs containing a certain note, so you can also use them to tag jobs.

Delayed jobs

  # Job will not be processed for 60 seconds
  $minion->enqueue('check_links', ['https://mojolicious.org'], {delay => 20});

You can use "retry" in Minion::Job to change the delay.

  $job->retry({delay => 10});

Expiring jobs

  # Job will vanish if it is not dequeued within 60 seconds
  $minion->enqueue('check_links', ['https://mojolicious.org'], {expire => 60});

You can use "retry" in Minion::Job to reset the expiration time.

  $job->retry({expire => 30});

Custom workers

  #!/usr/bin/perl
  use strict;
  use warnings;

  use Minion;

  # Connect to backend
  my $minion = Minion->new(Pg => 'postgresql://postgres@/test');

  # Add tasks
  $minion->add_task(something_slow => sub ($job, @args) {
    sleep 5;
    say 'This is a background worker process.';
  });

  # Start a worker to perform up to 12 jobs concurrently
  my $worker = $minion->worker;
  $worker->status->{jobs} = 12;
  $worker->run;

The method "run" in Minion::Worker contains all features you would expect from a Minion worker and can be easily configured with "status" in Minion::Worker. For even more customization options Minion::Worker also has a very rich low level API you could for example use to build workers that do not fork at all.

Task plugins

  package MyApp::Task::PokeMojo;
  use Mojo::Base 'Mojolicious::Plugin', -signatures;

  sub register ($self, $app, $config) {
    $app->minion->add_task(poke_mojo => sub ($job, @args) {
      $job->app->ua->get('mojolicious.org');
      $job->app->log->debug('We have poked mojolicious.org for a visitor');
    });
  }

  1;

Which are loaded like any other plugin from your application.

  # Mojolicious
  $app->plugin('MyApp::Task::PokeMojo');

  # Mojolicious::Lite
  plugin 'MyApp::Task::PokeMojo';

Task classes

  package MyApp::Task::PokeMojo;
  use Mojo::Base 'Minion::Job', -signatures;

  sub run ($self, @args) {
    $self->app->ua->get('mojolicious.org');
    $self->app->log->debug('We have poked mojolicious.org for a visitor');
  }

  1;

Task classes are registered just like any other task with "add_task" in Minion and you can even register the same class with multiple names.

  $minion->add_task(poke_mojo => 'MyApp::Task::PokeMojo');