Drupal 11: Batch Processing Using Drush

This is the second part of a series of articles looking at the Batch API in Drupal. The Batch API is a system in Drupal that allows data to be processed in small chunks in order to prevent timeout errors or memory problems.

In the previous article we looked at how to setup the batch process using a form, with the batch methods being contained within the structure of the form class. When the form was submitted the batch process ran through 1,000 items and then printed out a result at the end.

Whilst there is nothing wrong with running the Batch API with everything in a form class, it is normally better to abstract the batch processing code into a separate class.

Using a separate batch class to contain the process and finish methods is a much better way of setting things up as it allows you to abstract away the batch process from the action that starts it. This means that you can start the batch from anywhere, even a Drush command.

Allowing you batch processes to be run via Drush is a really powerful feature for a module to include. It means that any big process that can be run by a user can be run automatically via a Drush command.

The Batch Class

To create a batch class I normally create a directory called "Batch" inside the module "/src" directory that contains any batch class I need to define. The contents of the class are the two batch methods from the form class used previously, namely the batchProcess() and batchFinished() methods.

The following shows the basic structure of this class.

<?php

namespace Drupal\batch_class_example\Batch;

/**
 * Defines a process and finish method for a batch.
 */
class BatchClass {

  /**
   * Process a batch operation.
   *
   * @param int $batchId
   *   The batch ID.
   * @param array $chunk
   *   The chunk to process.
   * @param array $context
   *   Batch context.
   */
  public static function batchProcess(int $batchId, array $chunk, array &$context): void {
    // Process the batch here...
  }

  /**
   * Handle batch completion.
   *
   * @param bool $success
   *   TRUE if all batch API tasks were completed successfully.
   * @param array $results
   *   An results array from the batch processing operations.
   * @param array $operations
   *   A list of the operations that had not been completed.
   * @param string $elapsed
   *   Batch.inc kindly provides the elapsed processing time in seconds.
   */
  public static function batchFinished(bool $success, array $results, array $operations, string $elapsed): void {
    // Finish the batch here.
  }

}

I haven't changed the internal code of these two methods at all and so I have not reproduced them here. If you want to grab the code then you can either look at the previous Batch API article, or have a look at the full class on GitHub. All of the source code for this Batch API example and the other Batch API articles in this series are available on GitHub.

Running The Batch Class Via A Form

Now that we have the batch class in place we can rework the form submission handler to use this class. This is just a case of changing the setFinishCallback() and addOperation() methods to point at the new class.

Doing this changes the form submission handler that runs the batch running code to the following.

public function submitForm(array &$form, FormStateInterface $form_state): void {
  $batch = new BatchBuilder();
  $batch->setTitle('Running batch process.')
    ->setFinishCallback([BatchClass::class, 'batchFinished'])
    ->setInitMessage('Commencing')
    ->setProgressMessage('Processing...')
    ->setErrorMessage('An error occurred during processing.');

  // Create 10 chunks of 100 items.
  $chunks = array_chunk(range(1, 1000), 100);

  // Process each chunk in the array.
  foreach ($chunks as $id => $chunk) {
    $args = [
      $id,
      $chunk,
    ];
    $batch->addOperation([BatchClass::class, 'batchProcess'], $args);
  }
  batch_set($batch->toArray());

  $form_state->setRedirectUrl(new Url($this->getFormId()));
}

When we submit the form the batch operations run normally, the only difference here is where the batch operations are being run from.

Running The Batch From Drush

Now that we have separated the batch running code into a separate class we can run the same code from a Drush command.

To set up a Drush command we need to create a drush.services.yml file in the root of the module. This file looks very much like the normal module *.services.yml file that we use to create custom services in Drupal 8+.

The following code is added to the drush.services.yml file and will setup a Drush class that will inject commands into the Drush namespace. We also inject the logger.factory service so that we can log information in the command.

services:
  batch_class_example.commands:
    class: \Drupal\batch_class_example\Commands\BatchCommands
    tags:
      - { name: drush.command }
    arguments: ['@logger.factory']

The Drush class contains a single method (aside from the boilerplate code to inject the logger service) that defines the Drush command. We use some annotations to tell Drush that we have a command to define.

/**
 * Run a batch operation via a Drush command.
 *
 * @command batch_class_example:run
 *
 * @validate-module-enabled batch_class_example
 *
 * @usage batch_class_example:run
 */
public function runBatchclassExample() {
}

With this in place we can run drush batch_class_example:run and this method will be executed, so let's add our batch setup code.

To setup and run the batch we just need to copy the batch setup code from the form class. We are essentially running setting up the batch in the same way and the batch will be executed in a similar manner. The key difference when setting up the batch in Drush is that we also add a call to drush_backend_batch_process() function. This is an internal Drush function that will progressively process the batch to completion, instead of processing the batch via progressive page loads.

Here is the BatchCommands class in full.

<?php

namespace Drupal\batch_class_example\Commands;

use Drupal\batch_class_example\Batch\BatchClass;
use Drupal\Core\Batch\BatchBuilder;
use Drupal\Core\Logger\LoggerChannelFactoryInterface;
use Drupal\Core\StringTranslation\StringTranslationTrait;
use Drush\Commands\DrushCommands;

/**
 * Drush commands for the batch_class_example module.
 */
class BatchCommands extends DrushCommands {

  use StringTranslationTrait;

  /**
   * Logger service.
   *
   * @var \Drupal\Core\Logger\LoggerChannelFactoryInterface
   */
  private $loggerChannelFactory;

  /**
   * Constructs a new BatchCommands object.
   *
   * @param \Drupal\Core\Logger\LoggerChannelFactoryInterface $loggerChannelFactory
   *   Logger service.
   */
  public function __construct(LoggerChannelFactoryInterface $loggerChannelFactory) {
    $this->loggerChannelFactory = $loggerChannelFactory;
  }

  /**
   * Run a batch operation via a Drush command.
   *
   * @command batch_class_example:run
   *
   * @validate-module-enabled batch_class_example
   *
   * @usage batch_class_example:run
   */
  public function runBatchclassExample() {
    $batch = new BatchBuilder();
    $batch->setTitle('Running batch process.')
      ->setFinishCallback([BatchClass::class, 'batchFinished'])
      ->setInitMessage('Commencing')
      ->setProgressMessage('Processing...')
      ->setErrorMessage('An error occurred during processing.');

    // Create 10 chunks of 100 items.
    $chunks = array_chunk(range(1, 1000), 100);

    // Process each chunk in the array.
    foreach ($chunks as $id => $chunk) {
      $args = [
        $id,
        $chunk,
      ];
      $batch->addOperation([BatchClass::class, 'batchProcess'], $args);
    }
    batch_set($batch->toArray());

    drush_backend_batch_process();

    // Finish.
    $this->logger()->notice("Batch operations end.");
    $this->loggerChannelFactory->get('batch_class_example')->info('Batch operations end.');
  }

}

The final step in the batch process method is to send feedback to the user in the form of a message, and also to log that the batch finished.

We can now run the batch operation in Drush, which will print out the following messages.

$ drush batch_class_example:run
>  [notice] Processing batch #0 batch size 100 for total 1,000 items.
>  [notice] Processing batch #1 batch size 100 for total 1,000 items.
>  [notice] Processing batch #2 batch size 100 for total 1,000 items.
>  [notice] Processing batch #3 batch size 100 for total 1,000 items.
>  [notice] Processing batch #4 batch size 100 for total 1,000 items.
>  [notice] Processing batch #5 batch size 100 for total 1,000 items.
>  [notice] Processing batch #6 batch size 100 for total 1,000 items.
>  [notice] Processing batch #7 batch size 100 for total 1,000 items.
>  [notice] Processing batch #8 batch size 100 for total 1,000 items.
>  [notice] Processing batch #9 batch size 100 for total 1,000 items.
>  [notice] Message: Class batch completed processed 1000, skipped 272, updated 504, failed 224 in 
> 0 sec.
> 
 [notice] Batch operations end.

As in the previous article, this batch command is just running through 1,000 items and randomly deciding on the outcome of the run without making any changes to data on the system. This is deliberate so that the batch example can be run multiple times without adding or removing data from the Drupal site.

Conclusion

Batch operation in Drupal are useful in their own right, but by abstracting their processing and finish methods away from where they are setup we can run the batch operations from wherever we need to. Doing this gives users of the system the ability to automate tasks on the command line so that a user doesn't need to sit and wait for the batch processing page to complete.

This is normally how I set things up when using the Batch API.

We could even take this a step further by having a central place where the batch operations are set up, like a service for example, and then just call this service from the form and Drush command. This means we don't need to copy and paste any code when we want to kick off the batch processing.

In the next article in this series we will be looking at the finish state of the batch processor methods that gives us the ability to run the batch process for as long as we need. This is different to what I have currently shown where we setup the initial conditions of the batch run and let it execute to completion.

If you want to experiment and use this code in your own projects then you can find all of the source code for this module on GitHub. The source code is a sub module of a in a module that contains all of the source code for all of the articles in this series about the Drupal Batch API.