Drupal 10: Testing Migration Process Plugins

Drupal's migration system allows the use of a number of different plugins to perform source, processing, and destination functions. 

Process plugins are responsible for copying and sometimes manipulating data into the destination. There are a number of different process plugins that allow you to get data in different ways and and apply it to your destination fields.

Both the core Migrate module and the excellent Migrate Plus module contain a number of different process plugins that you can use to process your data in different ways.

Out of the box, the default process plugin is the get plugin, which can be used like this in your migration scripts.

destination_field:
  plugin: get
  source: source_field

This is often shortened to the following, which has exactly the same functionality.

destination_field: source_field

Most of the time you will want to avoid creating custom plugins, but sometimes your migration requirements will necessitate their use. You might find that your source data is very messy and needs to be cleaned up before importing it into the site. Process plugins are a really good way of doing this, but it is essential that you write tests to cover every situation that you might encounter. 

In this article we will look at two custom migrate process plugins that are built in different ways and how to test them. This will dive into some concepts around Drupal plugin management, dependency injection, as well as unit testing and data providers with PHPUnit.

First, let's look at the migration script that we will be using in this article. All of the source code for this migration example is available on GitHub.

We will use the embedded_data migrate source plugin so that we can embed the source data directly in the migrate script itself. This plugin is useful for very quick migrations, but also benefits us here as we can use it as a pedagogical device. The source data here is intentionally messy, to simulate a messy migration source.

id: migration_process_test
label: Testing custom migration process plugins

source:
  plugin: embedded_data
  data_rows:
    - data_id: 1
      data_title: '<p>About us page</p>'
      data_content: '<p>The about us page content.<span></span></p>'
    - data_id: 2
      data_title: '<p>contact Us page</p>'
      data_content: '<p>Contact us via the address [email protected].</p>'
  ids:
    data_id:
      type: integer

process:
  # Title field.
  title:
    - plugin: reformat_title
      source: data_title
  # Body field.
  body/0/value:
    - plugin: fix_data_content
      source: data_content
  body/0/format:
    plugin: default_value
    default_value: "basic_html"

destination:
  plugin: entity:node
  default_bundle: page

We are using two custom process plugins in the code above.

• reformat_title - The data we have from the source has the titles with markup in them, which we want to remove. Also, lots of pages in the source data have "page" in the title so we want to remove this word wherever it occurs. As a final step, the title should have the first letter of each word capitalized.

  This migrate plugin has no dependencies and so will be tested using a unit test setup.

• fix_data_content - Like many migrate sources, the source data for body fields is pretty messy. There are tags containing no text that we want to remove from the markup before it is added to the site. A number of hard coded contact email addresses have also been added to the markup, which we want to swap for the email address for our new site

  This migrate plugin requires the site configuration factory (config.factory) to be injected as a service so that we can get hold of the site email address. As this is a more involved plugin setup we will use a kernel test to test this plugin

We aren't using any configuration parameters for these process plugins in order to keep things as simple as possible.

The destination of this migration is the Page content type, which we get when using the Standard Drupal install profile.

Creating The Reformat Title Migrate Plugin

Creating a process plugin in migrate is quite easy (thankfully). Drupal plugin management system is used to pick up the plugin name using an annotation and generate the plugin object we can use.

In the migrate script we defined the reformat title plugin to have the id "reformat_title", so we need to create a class that extends \Drupal\migrate\ProcessPluginBase in the location src/Plugin/migrate/process.

We add the following annotation to this class to announce that it is a process plugin. All process plugin classes must contain this.

@MigrateProcessPlugin(id = "reformat_title")

The transform() method of the process plugin class, which is what the migrate system will execute during the migration, requires two objects from the migrate system. These are the current migrate system executable and an object representing the current row being processed.

The entire plugin class for the reformat_title plugin isn't that big, it just contains the annotation and the transform method.

<?php

namespace Drupal\migration_process_test\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;

/**
 * Reformat the title of the page.
 *
 * @code
 * title:
 *   plugin: reformat_title
 *   source: body/0/value
 * @endcode
 *
 * @MigrateProcessPlugin(id = "reformat_title")
 */
class ReformatTitle extends ProcessPluginBase {

  /**
   * {@inheritDoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    if ($value === NULL) {
      return $value;
    }

    // Strip any markup that the title might have.
    $value = strip_tags($value);

    // Strip any ending "page" words.
    $value = preg_replace('/\spage\s?$/', '', $value);

    // Make the string sentence case.
    $value = ucwords($value);

    return $value;
  }

}

The transform method here is just performing the changes that we want on the string coming from the source.

Unit Testing The Reformat Title Migrate Plugin

The reformat_title migrate process plugin doesn't require any additional dependencies and as such can be tested using a unit test.

In order to unit test a migrate process plugin we need to extend the class \Drupal\Tests\migrate\Unit\process\MigrateProcessTestCase. This class extends the Drupal core \Drupal\Tests\UnitTestCase class and adds some boiler plate code so that we can instantiate the plugin object without having to write our own mocking code.

As I mentioned earlier, the transform() method of the process plugin, which is what the migrate system will execute during the migration, requires two objects from the migrate system. These are the current migrate system executable and an object representing the current row being processed. The MigrateProcessTestCase object will create two properties that we can use to call transform() and not worry about where to get those objects from.

If your process plugin needs to pull some values out of the row object then you can use the global property to mock methods and return values as you normally would.

To create the process plugin as an usable object we just need to call it, the constructor (defined in \Drupal\Component\Plugin\PluginBase) requires some simple arguments to instantiate the object, but as none of them are objects we don't need to do any mocking.

$plugin = new ReformatTitle([], 'reformat_title', []);

With the plugin object in hand we can then call the transform() method, passing in the source value we want to test and the migrate executable and row arguments generated in the parent class. This means that our test can be boiled down to the following two lines.

$value = $plugin->transform('<p>About us page</p>', $this->migrateExecutable, $this->row, 'title');
$this->assertEquals('About Us', $value);

This is fine for a single value, but a much better way of running process plugin tests is using a data provider.

Data providers are PHPUnit plugins that will call our test case multiple times, each time with a different test setup. To set them up you just need to add the @dataProvider property to your test docblock comment, and define the method that will be used for the data provider.

This is the full unit test class for the reformat_title migrate process plugin, which lives in the test/Unit/Plugin/migrate/process directory within our migration module.

<?php

namespace Drupal\Tests\migration_process_test\Unit\Plugin\migrate\process;

use Drupal\migration_process_test\Plugin\migrate\process\ReformatTitle;
use Drupal\Tests\migrate\Unit\process\MigrateProcessTestCase;

/**
 * Tests the reformat_title migration plugin.
 */
class ReformatTitleTest extends MigrateProcessTestCase {

  /**
   * Test that different title values reformat correctly.
   *
   * @dataProvider titleIsReformattedDataProvider
   */
  public function testTitleIsReformatted($sourceValue, $expectedResult) {
    $plugin = new ReformatTitle([], 'reformat_title', []);
    $value = $plugin->transform($sourceValue, $this->migrateExecutable, $this->row, 'title');
    $this->assertEquals($expectedResult, $value);
  }

  /**
   * Data provider for testTitleIsReformatted.
   *
   * @return array
   *   The data to be tested.
   */
  public function titleIsReformattedDataProvider() {
    return [
      [
        '<p>About us page</p>',
        'About Us',
      ],
      [
        '<p>contact Us page</p>',
        'Contact Us',
      ],
    ];
  }

}

We can now test our process plugin with lots of different permutations of data.

Creating The Fix Data Content Migrate Plugin

The fix_data_content migrate process plugin has the same setup as the reformat_title plugin, with one exception. We want to inject a dependency into this plugin and so we can use a Drupal service.

As a side note, I actually struggled to find an example that would be simple enough to demonstrate the concepts involved, but not too complex that it required a bunch of other services and dependencies to get working. To this end I decided that a simple email address swap would be the simplest thing we could do. We just need to grab the site email from the Drupal configuration, which means injecting the config.factory service into the plugin.

The plugin.manager.migrate.process service is aware that we might want to have dependencies injected into the object and as such will look to see if the plugin class extends the interface \Drupal\Core\Plugin\ContainerFactoryPluginInterface. If it does then the plugin manager will call a static create() method within the class that will create the object and inject any dependencies.

In the case of this plugin we want to get access to a set of configuration, so we grab the config.factory service and extract the system.site configuration, which we store in a variable.

The full class for the fix_data_content migrate process plugin isn't that large. In fact, it mostly consists of the boilerplate code needed to setup the object with the dependencies we require.

<?php

namespace Drupal\migration_process_test\Plugin\migrate\process;

use Drupal\Core\Config\ImmutableConfig;
use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
 * Fix any broken markup in the source field.
 *
 * @code
 * body/0/value:
 *   plugin: fix_data_content
 * @endcode
 *
 * @MigrateProcessPlugin(id = "fix_data_content")
 */
class FixDataContent extends ProcessPluginBase implements ContainerFactoryPluginInterface {

  /**
   * The config factory object.
   *
   * @var \Drupal\Core\Config\ImmutableConfig
   */
  protected ImmutableConfig $siteConfig;

  /**
   * {@inheritDoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
    $instance = new self($configuration, $plugin_id, $plugin_definition);

    $instance->siteConfig = $container->get('config.factory')->get('system.site');

    return $instance;
  }

  /**
   * {@inheritDoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    if ($value === NULL) {
      return $value;
    }

    // Strip any empty elements.
    $value = preg_replace('/<span>\s*<\/span>/', '', $value);
    $value = preg_replace('/<p>\s*<\/p>/', '', $value);

    // Replace all instance of [email protected] with our site email.
    $value = preg_replace('/[email protected]/', $this->siteConfig->get('mail'), $value);

    return $value;
  }

}

The transform method here will use a couple of regular expressions to change the data being passed to the body field.

Kernel Testing The Fix Data Content Migrate Plugin

A kernel test differs a little from a unit test in that it allows you to minimally bootstrap Drupal. During the setup of the test a minimally installed Drupal site will be setup and you can then augment this with the modules, entities, and schemas that you need. This includes installing the module you are currently testing.

For example, if you wanted to test something with a User entity then you would install the user module, which makes sense. But you would also need to install the user entity schema in order to generate the tables needed for the entity to exist.

With regards to our fix_data_content migrate process plugin, because it uses Drupal configuration there it a little bit more setup to manage before we can get to actually running any tests. As Drupal is minimally bootstrapped we do have access to some of the core services within the system, one of which being the configuration system.

Kernel tests in Drupal extend the \Drupal\KernelTests\KernelTestBase class. When run, this class will automatically detect a class property called $modules, which it will use to determine the list of modules that must be installed. In our case, we just need the core migrate module, our own custom module (called migration_process_test) and the system module for the core system and configuration setup.

protected static $modules = [
  'migrate',
  'migration_process_test',
  'system',
];

It's generally a good idea to setup the conditions of your test in the setUp() method of the test, which is provided as part of the PHPUnit testing framework. This function is called automatically and so gives us a means by which we can setup Drupal to have everything we need before running the test. Make sure you call the setUp() method of the parent class first.

protected function setUp(): void {
  parent::setUp();
}

The setUp() method in our kernel test class needs to perform a couple of actions.

First, we need to setup the configuration so that the system.site.mail configuration setting contains a known value. To do this we just ask Drupal for the config.factory service and change the value we are going to use in the test.

// Update the site configuration with our test email address.
\Drupal::service('config.factory')->getEditable('system.site');
$system = $this->config('system.site');
$system
  ->set('mail', '[email protected]')
  ->save();

Next, because the KernelTestBase class doesn't extend the migrate MigrateProcessTestCase class we don't have the migrate executable and row objects that come for free in the unit test case. In which case we need to add a little bit of code to the setUp method in order to generate these objects as mocks.

// Create the test row and executable objects.
$this->row = $this->getMockBuilder('Drupal\migrate\Row')
  ->disableOriginalConstructor()
  ->getMock();
$this->migrateExecutable = $this->getMockBuilder('Drupal\migrate\MigrateExecutable')
  ->disableOriginalConstructor()
  ->getMock();

This is everything that is needed in the setUp() method so we can now go about generating our plugin object within our test case.

To generate the migrate process plugin in our test case we need to use the plugin.manager.migrate.process service. The createInstance() method of this plugin will automatically call the create() method in our plugin class, which we used to generate the dependencies we require for the plugin to work. In order to call the createInstance() method we need to supply a couple of extra properties. These are a migration object and any configuration we want to pass to the plugin itself.

The plugin.manager.migration service comes with a method called createStubMigration(), which we can use here to generate a simple test migration. This migration isn't actually going to be used here (this is a process plugin) so it just needs the bare minimum setup.

// Create migration stub.
$migration = \Drupal::service('plugin.manager.migration')
  ->createStubMigration([
    'id' => 'test',
    'source' => [],
    'process' => [],
    'destination' => [
      'plugin' => 'entity:node',
  ],
]);

// Set plugin configuration.
$configuration = [];

// Generate the plugin via the plugin.manager.migrate.process service.
$plugin = \Drupal::service('plugin.manager.migrate.process')
      ->createInstance('fix_data_content', $configuration, $migration);

The $plugin variable now contains a fully working instance of our plugin class, which we can call the transform() method on just like we did before.

$value = $plugin->transform('<p>Some text.<span></span></p>', $this->migrateExecutable, $this->row, 'field_body');
$this->assertEquals('<p>Some text.</p>', $value);

Of course, this is just a single test case, which should be abstracted out into a data provider test setup just like the unit test example.

The test/Kernel/Plugin/migrate/process directory within our custom migration module.

Mocking Services In Kernel Tests

Another consideration that we haven't touched on above is the ability to mock services in your kernel tests. This is useful when testing plugins as you can mock any calls to the database in order to return known data to your unit tests.

A common example of this is mocking the migrate.lookup service, which is used to lookup mapped values from the source data to the destination data. To mock this class we would do something like the following.

$migratePluginManager = $this->getMockBuilder('\Drupal\migrate\Plugin\MigrationPluginManager')
  ->disableOriginalConstructor()
  ->getMock();
$this->migrateLookup = $this->getMockBuilder('\Drupal\migrate\MigrateLookup')
  ->setConstructorArgs([$migratePluginManager])
  ->getMock();

We then need to inject this mocked version of the migrate.lookup service into the Drupal containers system. We do this by getting the container, setting the migrate.lookup to be our custom plugin and then setting the container object back into Drupal.

$container = \Drupal::getContainer();
$container->set('migrate.lookup', $this->migrateLookup);
\Drupal::setContainer($container);

After doing this our mocked migrate.lookup service will be used, which means that we can do things like this in our testing code.

$this->migrateLookup->method('lookup')->willReturn([['mid' => 1]]);

This way, we don't need to ensure that databases exist for mocked data, the mocked migrate.lookup can be made to return any value that we require.

Conclusion

Once you get used to setting up unit and kernel testing classes as required for your migrate plugins you can look at using test driven development to create them. Using test driven development helps to write the code you need without actually running a migration.

Testing your migrate process plugins is a very powerful tool when writing complex migration. Not only can it help you spot and correct edge cases in the source data, but it also can help speed up your migration development.

I have been involved in migrations on sites with LOTS of data; often involving lots of dependencies to look up field values. This can mean that in order to test the migration you are looking at hours of setup to get to the point where you can see your migration in action. By using unit tests for your process plugins you can quickly update your data provider with edge cases, work to solve them in your plugin, and commit your changes.

If you want the full source code of this module then it available on GitHub. Feel free to use it as a skeleton when setting up your custom migrate plugins.

If you have any comments or questions about migration plugins then please let me know in the comments below. Or, get in touch via the contact form for a more involved look.