14 minute read Published: 31 Aug, 2021 Author: Matt Parker
Drupal Planet , Migrations

Easy commit credits with migrations, part 4: Migrating D7 variables

This is the fourth in a series of blog posts on writing migrations for contrib modules:

In part 1, we set up a simple core migration.
In part 2, I covered how to review and manually tested patches.
In part 3, I demonstrated how to write automated migration tests
In this part, I’ll demonstrate how to migrate simple configuration (“variables” in D6 and D7 parlance).
In part 5, I explain how to declare a module’s migration status to the migrate wizard.
In part 6, I show how to migrate data from a custom database table.

Stay tuned for more in this series!

Background

While migrating off Drupal 7 Core is very easy, there are still many contrib modules without any migrations. Any sites built using a low-code approach likely use a lot of contrib modules, and are likely blocked from migrating because of contrib. But — as of this writing — Drupal 7 still makes up 60% of all Drupal sites, and time is running out to migrate them!

If we are to make Drupal the go-to technology for site builders, we need to remember that migrating contrib is part of the Site Builder experience too. If we make migrating easy, then fewer site builders will put off the upgrade or abandon Drupal. Plus, contributing to migrations gives us the opportunity to gain recognition in the Drupal community with contribution credits.

Problem / motivation

In my experience as a consultant, clients who are willing to be early adopters of Drupal 7 to 9 migrations tend to want to make a bunch of other changes to their site at the same time… so configuration has often been overlooked in favour of setting new config from scratch on the D9 site. But from an end-user-of-Drupal’s standpoint, when the budget is tight, and/or Drupal 7 already functions the way you want it, it makes more sense to spend your time and money on verifying the site’s content, and updating the website’s theme!

As a Site Builder migrating a site from Drupal 7 to Drupal 9, I want as much of my Drupal 7 configuration to be migrated as possible, so that I can spend my time on the theme and content of the site.

Proposed resolution

Define a migration for simple configuration from Drupal 7 to Drupal 9.

As mentioned briefly in the last post, migrations are defined by YAML files inside a module’s migrations/ directory that look something like…

id: MIGRATION_NAME
label: A Human-Friendly Name
migration tags:
  - Drupal 7
  - A Migration Tag
source:
  plugin: # a @MigrateSource plugin id
  # some config for that @MigrateSource plugin
process:
  # some process config
destination:
  plugin: # a @MigrateDestination plugin id
  # some config for that @MigrateDestination plugin

As you can probably guess, id, label, and migration tags are metadata.

Each migration definition includes a source plugin and its configuration, which states where to find data in the Drupal 7 source database. Each migration also defines a destination plugin and its configuration, which tells Drupal 9 where to store the migrated data. Each migration also contains a number of process instructions, which describe how to build the destination data, usually by taking data out of the source.

Steps to complete

Before we can write a simple config migration, we need to understand how config is stored in both systems; and do a bit of planning.

How Drupal 9 config works

In Drupal 9, the standard way to handle configuration is to store it as configuration entities, using the configuration management API.

Most configuration migrations into D9 use the config destination plugin, which uses the configuration management API to write the data. You configure the config destination plugin by specifying the machine name of the configuration entity that you want to build from the migrated data. If we look at the example migration we wrote tests for in the last blog post (i.e.: migrating config for the Environment Indicator module), you can see in its destination section…

destination:
  plugin: config
  config_name: environment_indicator.settings

… that the migration is going to be building the config object named environment_indicator.settings.

Note that each migration has one destination plugin; and the config destination plugin only lets you specify one config entity. To start a configuration migration, I usually look at the Drupal 9 module’s code for configuration objects. If there is more than one, I start with the one containing general configuration settings.

Once I’ve chosen a destination configuration object to focus on, I look at its definition in the module’s config/schema/MODULE_NAME.schema.yml file and where the config is being used (because the schema file isn’t always kept up-to-date). I start an inventory of the fields in that config object, their data type, and their default values from config/install/*.yml. A spreadsheet is a great tool for this inventory (just be aware that it can be helpful to show the spreadsheet to the community).

How Drupal 7 config works

In Drupal 7, the standard way to handle configuration was to store it in Drupal 7’s variable table in the database; and interact with it using the variable_get(), variable_set() and variable_del() functions.

My next step in writing a configuration migration is to search the D7 module’s code for the string variable_, examine the different variable names, and update my inventory with the D7 variable names and data types (to determine a variable’s data type, you may have to look at how the D7 code uses it). Some modules construct their variable names by concatenating strings and variables, so one string match (for variable_) may correspond with a handful of possible variables. If the way that variable names are constructed is particularly convoluted, it can be helpful to install the module on your D7 site, configure it, and see which variables are added to the variable table in the database.

When writing our migration definition for Drupal 7 variables, we can use the variable source plugin to pull data from the D7 variables table. You configure the variable source plugin by specifying a bunch of variable names to read data from; and optionally, specify which D7 module you’re migrating from (which is useful when you’re migrating config from a bunch of D7 modules into one D9 module).

If we look at the example migration we wrote tests for in the last blog post, you can see in its source section…

source:
  plugin: variable
  variables:
    - environment_indicator_integration
    - environment_indicator_favicon_overlay
  source_module: environment_indicator

… that the migration is going to copy data out of the environment_indicator_integration and environment_indicator_favicon_overlay variables.

Mapping out the migration

At this point, your inventory should contain the names, data-types, and default values for a bunch of D9 config object fields; plus the names and data-types for a bunch of D7 variables.

The next step is to process the inventory: for each D9 config object field, see if you can find a corresponding D7 variable, and mark the relationship in the inventory. It is always worth comparing how a config variable is used in both versions of the module, just in case it is unrelated but happened to be given a similar name. You should expect to find D7 config which does not have corresponding config in D9 and vice-versa. If you see D9 config that is related to the D7 config, but isn’t an exact copy (e.g.: a single value in D7 is the first value in an array in D9), add a note… we’ll talk about this shortly.

When you are done, your inventory might look like this…

D9 field	D9 field data type	D9 field default value	←	How to process	←	D7 variable	D7 data type	Notes
toolbar_integration	array	[]	←	(copy)	←	environment_indicator_integration	array	(n/a)
favicon	boolean	FALSE	←	(copy)	←	environment_indicator_favicon_overlay	boolean	(n/a)

At this point, you have enough information to start writing the migration test, which we covered in the previous blog post.

Migration process configuration

Now that we know how to migrate the data, we can write the process part of the migration configuration. Each instruction in the process section is a mapping (i.e.: hash, dictionary, object) whose name is the destination field. Inside the mapping is a list of migrate process plugins, to be run in order from first to last. The value after the final process plugin has run gets inserted into the destination field.

To get data from the source, you use the get migrate process plugin, which you configure by specifying which source fields to use…

process:
  toolbar_integration:  # i.e.: the destination field in the 'environment_indicator.settings' config object
    - plugin: get
      source:
        - environment_indicator_integration  # i.e.: the source field
  favicon:
    - plugin: get
      source:
        - environment_indicator_favicon_overlay

… this configuration copies the data in the environment_indicator_favicon_overlay variable from D7, performs no other processing on it (i.e.: because there are no other instructions), and inserts it into the favicon field in the environment_indicator.settings config object.

Since copying data from a source field to a destination field without any processing is so common, there is a short-hand for this particular case. For example,

process:
  favicon:
    - plugin: get
      source:
        - environment_indicator_favicon_overlay

… is equivalent to…

process:
  favicon: environment_indicator_favicon_overlay

After converting the plugin: get lines to the shorthand, your migration config should look identical to the sample one that I gave in the previous blog post. If you replace migration given last time, with the one you just finished building (don’t forget to set the migration id — it must match the filename and be in the $this->executeMigrations(['...']); line in your test). You can verify this is the case by running the test again.

Multiple process plugins

If you do need to modify the D7 data before it gets saved to D9, it is possible to add additional process plugins. For example, the following configuration would get the data stored in the source’s my_d7_label field, URL-encode it, convert that URL-encoded data to a machine name, then store the resulting data to dest_field…

process:
  dest_field:
    - plugin: get
      source:
        - my_d7_label
    - plugin: urlencode
    - plugin: machine_name

… put another way, given the data A name in source field my_d7_label, the data written to destination field dest_field would be a_20name (i.e.: A name -> A%20name -> a_20name).

If you are performing other processing steps, Core and many Contrib process plugins will allow you to take a shortcut by replacing the stand-alone get step by specifying a source in the first processing step. For example,

process:
  dest_field:
    - plugin: get
      source:
        - my_d7_label
    - plugin: urlencode
    - plugin: machine_name

… is equivalent to…

process:
  dest_field:
    - plugin: urlencode
      source: my_d7_label
    - plugin: machine_name

… but you may find it easier to keep the stand-alone get step until you’ve completed the whole migration and you are certain that it works.

When you write tests for a migration that involves process steps which modify data, the data you add in your test fixtures will be different from the data you verify at the end of the test — explicitly calling this out in a comment can be helpful to other people reading the test (or yourself 6 months later, when you’ve forgotten why).

Default values

Once a Drupal 7 module has been ported to D9 for the first time, that module’s D7 and D9 codebases diverge. Features added to the D9 version aren’t always backported to the D7 version for various reasons. As a result, when preparing your inventory, it’s not unusual to find that the D9 config object has fields for configuration which doesn’t exist in D7 (note the converse — where D7 variables have no D9 equivalent — is possible too, albeit less common).

When you’re writing a migration for the first time, it’s easy to focus on the config that you can migrate from D7, and ignore the D9 config which has no D7 equivalent. But if you don’t specify a value for those fields, the config destination plugin will set them to NULL when it constructs the config object from the migrated data.

But, many modules assume that those configuration object fields will be set to their default configuration (i.e.: from config/install/*.yml) — not NULL — which can lead to bugs, errors, warnings, and crashes later on.

The solution is to specify default values for that configuration in the process section of the migration definition, using the default_value process plugin.

For example:

process:
  d9_only_feature:
    - plugin: default_value
      default_value: 'some_default_value'

When you specify default values, you should still test them, by verifying them when you verify the migrated data. I tend to separate these into their own section with a comment, so that I don’t get confused about why those verification lines don’t have a corresponding fixture…

// Verify the fixtures data is now present in the destination site.
$this->assertSame(['toolbar' => 'toolbar'], $this->config('environment_indicator.settings')->get('toolbar_integration'));
$this->assertSame(TRUE, $this->config('environment_indicator.settings')->get('favicon'));

// Verify the settings with no source-site equivalent are set to their default values in the destination site.
$this->assertSame('some_default_value', $this->config('environment_indicator.settings')->get('d9_only_feature');

Putting it all together

Make sure your Drupal 9 environment is set up as described in the last blog post:
1. Clone Drupal core, run composer install, set up the site.
2. Find a migration issue and module to work on.
3. Clone the module to modules/, and switch to the branch in the migration issue’s Version field if necessary.
4. If the migration issue is using an issue fork, then switch to the issue fork using the instructions in the issue.
5. If the migration issue is using patches, download the patch, apply it to a branch named after the issue ID and comment number the patch was uploaded to (we’ll call this $FIRST_BRANCH below).
6. If the migration issue is using patches, then create a second branch named after the issue ID and number of comments in the issue plus 1; and apply the patch, but don’t commit it yet.
Create the migration inventory.
Write the migration test.
Write the migration itself, running tests frequently.
Spin up your D7 site, install the D7 version of the module, and run a manual test, as we did in part 1 of this series.

The automated tests only test for very specific problems on a simulated “clean” environment — which make them great for catching regressions — but not very good for catching problems you weren’t specifically testing for (that is to say, things likely to crop up in the real world).

Note that Drupal 9 core’s Migrate Drupal UI module has no way of knowing if you’ve written all the migrations that you intended to write for this module. So, the module you’ve written the migration for will still show up in the list of “Modules that will not be upgraded” for now — we’ll fix that in the next blog post. Don’t worry though, your migration will still run.
When you’re satisfied, stage all the changes to the module (i.e.: git add .), and commit your changes. In the commit message, describe what you did. The commit message will be visible to other members of the community.
If the migration issue is using an issue fork, then push your changes to the issue fork, and leave a comment in the issue describing what you did.
If the migration issue is using patches, then:
1. Generate the patch with git format-patch 8.x-2.x (where 8.x-2.x is the branch specified in the “Version” field of the issue with the patch).
  
  Generating a patch in this way way adds some metadata which will help avoid merge conflicts in the future.
  
  The patch will appear in the current directory, and will be named something like 0001-YOUR-COMMIT-MESSAGE.patch.
2. Rename the patch according to Drupal.org’s conventions for naming patches.
  
  I like to move the patch somewhere that I can easily find it (e.g.: my Desktop folder) at the same time that I’m renaming it.
3. Generate an interdiff between your current patch and the previous one with git diff $FIRST_BRANCH > interdiff.txt (where $FIRST_BRANCH is the branch with the previous patch applied and committed).
  
  I like to move the interdiff somewhere that I can easily find it (e.g.: my Desktop folder).
4. Start a new comment, upload the patch and interdiff, and describe what you did in the Comment.
  
  If you set the issue status to “Needs review”, then automated tests will run on your patch — but once they pass, change the issue status back to “Needs work”, because your migration won’t be finished until you’ve verified there’s nothing else to migrate, and indicated that to the Migrate Drupal UI module.

If you’ve been following along with our example to migrate configuration for the Environment Indicator module, please be aware that there’s already a migration to do that in issue #3198995 — so please do not create a new issue, and please do not leave patches in that issue.

Next steps

At this point, you should have all the tools that you need to contribute patches which migrate simple configuration from the Drupal 7 version of a module to the Drupal 9 version of the module, so try it out!

Next, we will talk about how to tell Drupal core’s Migrate Drupal UI module that you’ve written all the migrations that you intended to write, which will move the module from “Modules that will not be upgraded” to “Modules that will be upgraded” in the migration wizard.

We’ll also talk about how to migrate more complex configuration and content in future posts in this series.

The article Easy commit credits with migrations, part 4: Migrating D7 variables first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

We help small teams do big things.