Drupal 11: Creating Custom Queues

Creating queues using the core queue classes in Drupal is fairly straightforward. You just need a mechanism of adding data to the queue and a worker to process that data.

As the data you add to the queue is serialised you can add pretty much any data you want to the queue, so the only limitation is rebuilding the data once you pull it out of the queue.

There are some situations where the core Drupal queue system needs to be altered in some way. You might want to separate the data into different tables, or have a different logic for creating or storing the queue items, or even integrate with a third party queue system for manage the queues.

Whilst all of these examples are possible, they require a certain amount of understanding of the queue API and need additional services and settings to get working.

In this article we will look at how to create a custom queue, along with the queue factory needed to integrate that queue with Drupal. We will also look at some settings needed to swap out certain queues for you custom queue implementations. All of the code seen in this article is available in our Drupal Queue Examples repository on GitHub, specifically the queue_custom_example module.

First, let's look at what is requires for a queue to work in Drupal.

Create A Custom Queue With The QueueInterface Interface

The interface \Drupal\Core\Queue\QueueInterface is used to build the framework of the queue, which is used to manage the queue items. Your queue object must have the following methods. 

• createItem($data) - Creates an item in the queue. The data you pass is serialised in Drupal core, but you can store the data in any way you want.
• numberOfItems() - Reports on the number of items in the queue.
• claimItem($lease_time = 3600) - Claim an item from the queue. The $lease_time variable is used to update the queue item so that it doesn't get claimed again within that time limit. The default being 3600 seconds, or 1 hour.
• deleteItem($item) - Delete an item from the queue. The $item here is the result of the claimItem() method and should have an identifier that can be used to target the queue item.
• releaseItem($item) - Release an item in the queue that was previously claimed. Again, the $item here is the return value from the claimItem() method. This should essentially reset the lease time back to 0 and the item will be picked up when it reaches the top of the queue.
• createQueue() - Create the storage for the queue by creating a custom queue table or whatever is needed to generate the queue. This is not used in Drupal core currently as the Memory queue doesn't need storage and the DatabaseQueue creates the table dynamically in the createItem() method.
• deleteQueue() - Remove the storage of the queue from the system.

Unless you are building a queue around an entirely custom queue handler then the best thing to do is to extend one of the existing core queue classes. This means that you don't need to implement all of these methods from scratch.

For example, the following CustomQueueDatabase class extends the core \Drupal\Core\Queue\DatabaseQueue class in order to change the name of the table in use. This queue will act just like the standard database queue, but will store the data in a table called "queue_custom". The rest of the functions above are included via the extending of the base class.

<?php

declare(strict_types=1);

namespace Drupal\queue_custom_example;

use Drupal\Core\Queue\DatabaseQueue;

/**
 * A custom queue based off of the core database queue.
 */
class CustomQueueDatabase extends DatabaseQueue {

  /**
   * The database table name.
   */
  public const TABLE_NAME = 'queue_custom';

}

We can then extent any functions in order to change how the underlying functionality of the queue works.

Alternatively, if we then wanted to integrate with a queue system like RabbitMQ then we would need to create a queue handler from scratch. That class would integrate with the RabbitMQ queue system and allow Drupal to manage items in that queue.

Create A Custom Queue Factory

In order to make use of this custom queue we need to create a queue factory. This is how Drupal creates the queue handler class internally and the easiest way to plug in the custom queue with the queue ecosystem in Drupal

First, we need to create a named service called "queue_custom_example.database". Since Drupal 11, the "autoconfigure" option in the services file is important as it tells Drupal that the parents services should be included into this service too.

services:
  _defaults:
    autoconfigure: true

  queue_custom_example.database:
    class: Drupal\queue_custom_example\CustomQueueDatabaseFactory
    parent: queue.database

The service is pretty simple, we have already told Drupal that we will extend the QueueDatabaseFactory class and so this is what we do in the class. The get() method needs to return an instance of the \Drupal\Core\Queue\QueueInterface interface. This is what we do when we create a new object of CustomQueueDatabase, which implements this interface, and is defined in the previous step above.

<?php

declare(strict_types=1);

namespace Drupal\queue_custom_example;

use Drupal\Core\Queue\QueueDatabaseFactory;

/**
 * A queue factory class to create the CustomQueueDatabase queue.
 */
class CustomQueueDatabaseFactory extends QueueDatabaseFactory {

  /**
   * {@inheritdoc}
   */
  public function get($name) {
    return new CustomQueueDatabase($name, $this->connection);
  }

}

There are two ways in which we can use this queue. The simplest way is to just use our custom queue service directly, which will return a usable queue item.

/** @var \Drupal\Core\Queue\QueueInterface $queue */
$queue = \Drupal::service('queue_custom_example.database')->get('queue_custom_example');

Alternatively, we can tell Drupal that when the "queue_custom_example" is used that a specific service class must be used to manage the queue. We do this by altering the queues using the settings.php files.

Altering Queues Using Settings

The following values can be set in your settings.php file's $settings array to define which services are used for queues:

• queue_reliable_service_$name - The container service to use for the reliable queue "$name". A reliable queue is any queue implements the \Drupal\Core\Queue\ReliableQueueInterface interface, and is expected to store values between page loads. The database queue is a reliable queue as it implements this interface.
• queue_service_$name - The container service to use for the queue "$name".
• queue_default - The container service to use by default for queues without overrides. This defaults to 'queue.database'.

As mentioned, these settings can be added to the site's settings.php file. For example, to change the service class used for the queue_custom_example queue we would do the following.

$settings['queue_service_queue_custom_example'] = 'queue_custom_example.database';

Make sure you clear the caches if you add this setting after installing the site.

Now, you can use the normal "queue" service from Drupal, but in this case the setting will cause the "queue_custom_example.database" factory to be used to load our custom queue class.

/** @var \Drupal\Core\Queue\QueueInterface $queue */
$queue = \Drupal::service('queue')->get('queue_custom_example');

If you want, you can use the "queue_default" setting to change all the queue handlers in your site to point at a custom service. It is advisable to leave the core queue handlers alone and just change the queue .

Note that this functionality was scheduled to be deprecated, but due to delays in getting the replacement feature ready it won't be deprecated until at least 11.1.0. The replacement functionality hasn't been decided on yet so I think its safe to document here. Check on the progress of this issue if you want to see when this change will make it into Drupal core.

Conclusion

The queue system in Drupal is highly modular and can even be swapped in and out as required. The default queue classes in Drupal work extremely well, but you can easily use a custom service for a specific queue when you want to extend and change that core functionality.

I remember in Drupal 7 having to override a lot of the built in queue functionality to get things working how I wanted. I haven't had to do the same with the queue system in Drupal 8+, which is a good sign. If you do need to alter things though, then it's fairly easy to plug in your custom queue handlers without having to write lots of code to do it.

The Queue Unique module creates a queue that enforces all queue items to be unique, and this module makes use of the $settings array in the settings.php file to enforce which queues should be treated as unique. The RabbitMQ module does the same thing by providing an interface with RabbitMQ via the Drupal queue system.

All of the code seen in this article is available in our Drupal Queue Examples repository on GitHub, specifically the queue_custom_example module. The queue custom example module shows both the direct queue service and the $settings queue name update being used to manage items in a queue.