https://consensus.enterprises/blog/ Recent content on the Consensus Enterprises Blog. Hugo -- gohugo.io en-CA info@consensus.enterprises (The Consensus Team) tech@consensus.enterprises (Consensus Infrastructure) Copyright 2025 Consensus Enterprises International Inc. Wed, 05 Feb 2025 09:00:00 -0500 https://consensus.enterprises/images/consensus-blog-banner.png https://consensus.enterprises/blog/ https://consensus.enterprises/blog/gitlab-automating-ticket-creation/ Wed, 05 Feb 2025 09:00:00 -0500 Automating Gitlab Issue Creation for Weekly Odoo Grooming on Consensus Enterprises Blog published Wed, 05 Feb 2025 09:00:00 -0500 At Consensus Enterprises, we use GitLab boards for task management and Odoo as our CRM for business development. Like any organizing tool, one needs to put in the time to keep it structured and actionable or the data within can get stale quickly. As a lean team with swappable roles, we prioritize efficiency and consistent workflow automation so we leveraged GitLab CI/CD to automate issue creation for our Odoo grooming process. Here’s how we built an automated pipeline to generate a structured … At Consensus Enterprises, we use GitLab boards for task management and Odoo as our CRM for business development. Like any organizing tool, one needs to put in the time to keep it structured and actionable or the data within can get stale quickly. As a lean team with swappable roles, we prioritize efficiency and consistent workflow automation so we leveraged GitLab CI/CD to automate issue creation for our Odoo grooming process. Here’s how we built an automated pipeline to generate a structured issue every week. Manually creating a GitLab issue each week was inefficient and prone to oversight. Our goal was simple: automatically generate a GitLab issue every Monday with a structured checklist to ensure our team reviews stalled, new, and ongoing business development opportunities. We set up a GitLab pipeline that: Creates a structured GitLab issue every week before our meeting. Assigns the correct users dynamically. Formats the issue description to maintain readability and structure. Runs on a schedule, ensuring it only triggers at the right time. Here’s how we did it. The core concept is straightforward: Create a standard .gitlab-ci.yml file in your repository. Write a script within the pipeline that uses cURL to post to your project via the GitLab API Schedule it using Gitlab’s scheduled pipelines You’ll need to also: Create a personal access token with the API scope Add it as a CI/CD variable and ensure it is masked for security Here’s what the .gtilab-ci.yml looks like: Note: The Gitlab pipeline editor is your friend GitLab CI/CD runs in containers, so we specified a Debian image and ensured jq and curl were installed: image: debian:latest # Ensures jq is available before_script: - apt-get update && apt-get install -y jq curl jq processes JSON responses from the GitLab API, filtering and transforming the data to extract relevant fields like user IDs for assignees. It also safely formats multi-line text for the issue description, escaping characters as needed. Finally, it constructs the JSON payload for the API request, ensuring correct syntax and preventing errors. We could just have a simple static title, but no one wants to manage a board of identical ticket names! So we dynamically generate a formatted title and set the due date for the following Tuesday: script: - export DAY_NUM=$(date '+%-d') - | case "$DAY_NUM" in 1|21|31) DAY_SUFFIX="st" ;; 2|22) DAY_SUFFIX="nd" ;; 3|23) DAY_SUFFIX="rd" ;; *) DAY_SUFFIX="th" ;; esac export DAY_SUFFIX - export TITLE="Odoo Grooming - $(date '+%B') ${DAY_NUM}${DAY_SUFFIX}, $(date '+%Y')" - export DUE_DATE=$(date -d 'next Tuesday' '+%Y-%m-%d') We prefer using checkboxes for clarity, but as long as you maintain the syntax structure, go ahead and modify the text as needed. The issue description needs to be multi-line and properly formatted for GitLab’s markdown rendering. Using jq, we embed it safely: - export DESCRIPTION="$(jq -n --arg desc \ "These tasks ensure our pipeline remains healthy and organized. Allocate two hours to this work before our weekly meeting.\n\n\ - [ ] Review \`STALLED\` items.\n\ - [ ] Mark \`LOST\` items (Define criteria for lost deals)\n\ - [ ] Update/Create new \`Activities\` (Prioritize moving stalled items)\n\ - [ ] Review \`NEW\` items\n\ - [ ] Mark \`LOST\` items\n\ - [ ] Update stages\n\ - [ ] Update/Create new \`Activities\`\n\ - [ ] Review all other opportunities (\`Prospects\`, \`Qualified\`, \`Proposal\`, \`Negotiation\`)\n\ - [ ] Mark \`LOST\` or \`WON\` items\n\ - [ ] Add log notes where necessary\n\ - [ ] Update stages\n\ - [ ] Update/Create new \`Activities\`" '$desc')" In the JSON payload we want to give a properly formatted array of assignees by their ID. eg. [1234567,1234567] We could just write these directly into the script but wanted to be slightly more dynamic here and validate the users are on the project and get their proper IDs. First we pull out the IDs by user name. Assignees need to be correctly retrieved from the GitLab API. Users added at the group level are not included in the project members list. We needed to fetch them from the right endpoint: - | export ASSIGNEES="$(curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \ "https://gitlab.com/api/v4/groups/YOUR_GROUP_ID/members/all" | \ jq '[.[] | select(.username=="USER_NAME_1" or .username=="USER_NAME_2") | .id]')" To ensure proper formatting and avoid breaking the script, we check if ASSIGNEES is empty and set it to an empty JSON array if necessary: - | if [ -z "$ASSIGNEES" ] || [ "$ASSIGNEES" == "[]" ]; then export ASSIGNEES="[]" fi We then construct the JSON payload and send the issue creation request: - | export JSON_PAYLOAD="$(jq --compact-output --null-input \ --arg title "$TITLE" \ --arg desc "$DESCRIPTION" \ --arg labels "Odoo, State::01 - To Do" \ --argjson assignees "$ASSIGNEES" \ --arg due_date "$DUE_DATE" \ '{ "title": $title, "description": $desc, "labels": $labels, "assignee_ids": $assignees, "due_date": $due_date }')" - | curl --request POST \ --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \ --header "Content-Type: application/json" \ --data "$JSON_PAYLOAD" \ "https://gitlab.com/api/v4/projects/YOUR_PROJECT_ID/issues" To ensure the job only runs when scheduled, we use: rules: - if: '$CI_PIPELINE_SOURCE == "schedule"' Note: this could also be written as a script in your repo that the CI/CD can run. Once you are done with the file you can schedule it using Gitlab’s scheduled pipelines Fetching Assignees Requires the Right Endpoint – Users added at the group level won’t appear under projects/{id}/members, and vice versa. Ensure you fetch from the correct location. jq Is Essential for JSON Handling – Since GitLab’s API expects properly formatted JSON, jq is essential for building and escaping multi-line descriptions. Use a Debian Image and Install Dependencies The default GitLab runner image may not include jq or curl, so specify Debian and install them in before_script. Automating issue creation in GitLab ensures our Odoo CRM grooming stays on track without requiring manual input. By dynamically generating structured issues, assigning the right users, and running on a schedule, we’ve eliminated a manual task and improved our workflow. Setting up a similar automation in GitLab CI/CD is worth the investment for teams looking to streamline their project management. https://consensus.enterprises/blog/pure-ansible-aegir3/ Wed, 11 Dec 2024 09:00:00 -0500 make targets, Droplets, and Aegir, oh my! on Consensus Enterprises Blog published Wed, 11 Dec 2024 09:00:00 -0500 We’ve done it! Over the last 6 posts, we’ve reviewed in detail all the major components of our simplified Infrastructure-as-Code (IaC) setup, and we’re ready to pull it all together to spin up an Aegir3 instance. With the pieces we have covered so far, we can run the following to get up the point of actually installing Aegir: pipenv shell # Activate venv source d # Bootstrap Drumkit make tools # Install Ansible and Galaxy dependencies make infra # Run playbooks/infra/00-up.yml This will get us a … We’ve done it! Over the last 6 posts, we’ve reviewed in detail all the major components of our simplified Infrastructure-as-Code (IaC) setup, and we’re ready to pull it all together to spin up an Aegir3 instance. With the pieces we have covered so far, we can run the following to get up the point of actually installing Aegir: pipenv shell # Activate venv source d # Bootstrap Drumkit make tools # Install Ansible and Galaxy dependencies make infra # Run playbooks/infra/00-up.yml This will get us a Project, a VPC, and a Firewall at DigitalOcean, configured with the names and settings we’ve described in our previous posts. To get from here to a working Aegir, all we need is a host Droplet, standard LEMP stack dependencies, and then Aegir itself. First of all, let’s use our consensus.infra.droplet role to create our host Droplet, and wrap a make aegir-droplet target to run the playbook playbooks/hosts/aegir0.yml: --- - name: aegir Droplet hosts: localhost gather_facts: true vars: droplet_hostname: aegir0 droplet_domain: cloudcity.dev droplet_project: "{{ do_project_name }}" droplet_firewall: "{{ web_droplet_firewall }}" droplet_tags: ["web", "aegir"] droplet_size: "s-2vcpu-2gb" droplet_vpc: "{{ do_vpc_name }}" outgoing_email_password: "{{ vaulted_outgoing_smtp_password }}" outgoing_email_default: "{{ org_technical_contact_email }}" oauth_token: "{{ vault_do_api_token }}" gandi_token: "{{ vaulted_gandi_access_token }}" roles: - consensus.infra.droplet As discussed previously, the Droplet-specific vars are embedded directly in the playbook here, because we don’t have a manifest or other registry to logically place these. This playbook (and others in playbooks/hosts) become the de facto manifest for the project. Now that we have a host, we can target it with a playbooks/apps/aegir.yml playbook to call our consensus.aegir.aegir role, and introduce our make aegir-app target. This playbook has some conditional checks before and after to ensure everything works correctly, but ultimately boils down to this: --- - name: "End-to-end Aegir 3 build from git source." hosts: aegir gather_facts: true roles: - role: geerlingguy.mysql become: True - role: geerlingguy.nginx become: True - role: geerlingguy.php-versions become: True - role: geerlingguy.php become: True - role: geerlingguy.composer become: True - role: consensus.aegir.aegir become: True We simply run a series of roles in sequence, building up the stack of software to support Aegir, and then finally Aegir itself. Tying the two previous playbooks together, we can implement a make aegir target simply: aegir: aegir-droplet aegir-app aegir-droplet: ansible-playbook playbooks/hosts/aegir0.yml aegir-app: ansible-playbook playbooks/apps/aegir.yml Added to the steps at the top of this post, this new target will complete the provisioning of an Aegir3 instance at DigitalOcean. Note that drumkit/mk.d/30_aegir.mk also has “down” variants of these targets as well: make aegir-down - deregister from Tailscale and destroy the Droplet make aegir-ts-down - only deregister from Tailscale (do not destroy Droplet) make aegir-droplet-down - only destroy the Droplet (in case Tailscale is already offline) A note about SSL: you will note that the playbooks and roles involved here DO NOT provision SSL certificates of any kind for the Aegir instance. I’ve left this piece out here for two reasons: There are many ways to achieve this, and we don’t need it for our purposes Even with SSL, your Aegir3 instance shouldn’t be exposed to the public internet anyway, since it’s based on Drupal 7. NB Please take care to properly secure your Aegir3 instance, as running Drupal 7 which is EOL! This could be by adding firewall rules, or reconfiguring your Nginx virtualhost to listen on your Tailscale interface, or similar. Also, if you’re not aware, Aegir5 is our effort to modernize Aegir based on modern Drupal with an Ansible-driven backend. This will be a secure replacement for Aegir, but since it’s not production-ready yet, we’re using Aegir3 as a stopgap for the time being. Over the years, we at Consensus have wrangled with a lot of necessary complexity. The nature of the problems we tend to help our clients solve often means it’s unavoidable. In that context it’s especially useful to stay focused on keeping things as simple as possible, ensuring we don’t make things worse by piling extra complexity on top of the inevitable stuff. The process of simplifying our internal infrastructure has been a great example of radical simplification. We’ve already seen a reduction in costs, both financial and operational. Our infrastructure team is reinvigorated by the ease of interacting with our new suite of servers, inspired to use and extend them. To review: We dropped Terraform, which is a great tool but represented unnecessary complexity in our context. We replaced a custom-built Wireguard server with Tailscale. We split our DEV and PROD environments into distinct repositories. We published key components of the solution to share between DEV and PROD. We built up a simple codebase structure to house infrastructure, host, and application playbooks. We used Drumkit to provide easy target commands to accomplish key workflow tasks. I hope this series has illustrated our approach and how it all fits together. If you are interested in exploring further, feel free to use our example repository as a base. We’ve left some detailed usage notes in the README with details on how to adapt it for your own use. If you do try something like this yourself, or you just have questions or comments, don’t hesitate to get in touch. For those in the Drupal community familiar with the Aegir project, I will say again: the use of Aegir3 here is a stopgap until we can get Aegir5 ready to replace it. In the meantime, we’re very pleased to have worked out a path for existing Aegir3 installations to support Drupal 8 through 11. Stay tuned for upcoming details on this as well as new developments with Aegir5! https://consensus.enterprises/blog/pure-ansible-collections/ Mon, 09 Dec 2024 09:00:00 -0500 Building Ansible Collections on Consensus Enterprises Blog published Mon, 09 Dec 2024 09:00:00 -0500 Initially our pure ansible infrastructure project consisted primarily of Ansible playbooks, using includes and imports to segment out what was essentially one long sequential process. We built up a set of plays that would go from nothing through all of the support infrastructure and provisioning a Droplet, up to the point of being ready to install application-level software, namely Aegir and its dependencies. At this point, we were able to pick up and dust off some roles that we had built over … Initially our pure ansible infrastructure project consisted primarily of Ansible playbooks, using includes and imports to segment out what was essentially one long sequential process. We built up a set of plays that would go from nothing through all of the support infrastructure and provisioning a Droplet, up to the point of being ready to install application-level software, namely Aegir and its dependencies. At this point, we were able to pick up and dust off some roles that we had built over the last number of years while supporting clients with Aegir installations. The 2 key roles here were consensus.aegir and consensus.aegir-minion, and while updating and refreshing these for current use, were able to further simplify things, eliminating a number of custom forks of upstream roles etc. At the same time, we updated the publishing of these roles to align with the new best practice of rolling them into collections (deprecating and archiving the originals). Having gone this far, we realized it would be straightforward to go one step further and roll our consensus.droplet role into a collection we could publish as well. This is arguably overkill in terms of simplicity, as we could have moved things into custom roles within the project quite easily. Two reasons convinced us to keep going. First, our commitment to open source: anywhere possible, we try to publish and share our work in hopes that it can benefit others. Second, sometimes we help our future selves by properly documenting and publishing things in this way. In this case, having 2 different IaC repositories means that if we stabilize and externalize the custom code within each, we have less to manually re-implement: we can share code outside the IaC repos themselves. Without further ado, let me introduce our shiny new Ansible collections: consensus.infra consensus.aegir These collections form the basis for the rest of our pure ansible infrastructure solution. Each collection contains a couple of roles, so let’s review each in turn. As alluded to previously, this collection contains a role that was born of our initial work building linear playbooks to: create a DigitalOcean Droplet, provision a Block Storage Volume, install a DNS A record at Gandi.net, join our Tailscale VPN, configure outbound email, install standard OS packages (Consensus flavour), restrict SSH access to the tailnet, and create administrative users with SSH access to the droplet. In fact, this last step is covered by the other role in this collection, admin-users. This role has been around for a number of years, and simply takes a list of users and SSH public keys, then creates those users on the target host, ensuring they have a .authorized_keys file to allow them to log in with their SSH key. Because these two are so closely linked for our purposes, we packaged them together into one collection. While these roles are published and available for anyone to use, I will note that the consensus.infra.droplet role is still very fresh, and thus very opinionated. The set of steps I outline above is configurable in that you can set a custom hostname and domain, and so on. However, you can’t skip any steps, or change out your DNS provider, for example. If your needs don’t fit exactly the steps this role provides, you’ll need to adapt it to your needs. Feel free to submit an MR on the project or fork your own copy to customize! Our second collection provisions the Aegir3 hosting system. As mentioned, these roles have been around a lot longer, and only needed some updates (for Ansible syntax, and newer OS releases) to be useful. We were also able to eliminate forks of a number of geerlingguy roles that we had previously customized to accommodate Aegir’s needs. Many of these customizations were no longer needed or incorporated upstream. We found ways to move those that remained directly into the Aegir role. This means we can now use only on the stock geerlingguy roles, and everything works fine. A small caveat: in our focus on simplicity, we’ve narrowed the scope to exactly the OS and version we’re targeting at present, namely Ubuntu 24.04. We know these roles work well in that context, and should be reasonably functional for 22.04 and even 20.04, with the possible need for some tweaking of PHP versions. If you’re targeting a Debian-like system, things are likely to work more or less as-is, but other Linux distributions are unlikely to function out of the box. That said, adapting to different distros and versions is usually a matter of adding some conditional flags that check the relevant Facts on the target host, and switch out the play they run appropriately. As always, we welcome patches! If you get this role running in a new environment, please submit an MR on the project to share your work :) Since we had now found ourselves with not one but two Ansible collections we wanted to publish (and more we can imagine wanting in future), we knew it would be wise to automate publishing to Galaxy. With the solo roles previously, our publishing workflow had been automatic but somewhat opaque and fragile. We prefer to work in GitLab, but Galaxy can only publish roles via GitHub. So we had set up mirroring of the repository to GitHub, and thus a push to the main branch in GitLab would result in a new package on Galaxy. With the move to Collections, it’s possible to publish directly to Galaxy using a couple of commands on the shell. This meant we could eliminate the GitHub mirror, which was the fragile part, and implement a CI pipeline on GitLab that would push out new release packages when we set a new git tag on the main branch. After some research, I found a post by Jeff Geerling himself describing the process he uses for this. I also found a couple of other posts that helped me adapt this to a GitLab context, and ended up producing a template project with the minimal pieces in place to auto-publish an Ansible Collection to Galaxy: publish/publish-collection.yml: an Ansible playbook to build and publish the collection. .gitlab-ci.yml: calls the playbook if: there is a git tag on the commit publish/galaxy.yml.j2: template metadata file for your collection meta/runtime.yml: boilerplate metadata README: detailed instructions on how to adapt the template for your use :) We have now reviewed all the major components of our pure ansible infrastructure project, and we’re ready to pull it all together. In our next post we’ll take a look at the final pieces, and take a step back to review the bigger, simpler picture. https://consensus.enterprises/blog/pure-ansible-infra/ Fri, 06 Dec 2024 09:00:00 -0500 Playbooks and support infrastructure on Consensus Enterprises Blog published Fri, 06 Dec 2024 09:00:00 -0500 Ansible Playbooks are the heart of any project driven by this tool, and in our example project we are now ready to look at this core piece of the puzzle. Essentially, when we think about the infrastructure resources needed to host a site or application like Aegir, there are 3 main categories to think about: Support infrastructure: VPCs and Firewalls, etc. Hosts: usually VMs, sometimes clusters or “bare metal” servers Applications: the software stack from operating system up through the hosted … Ansible Playbooks are the heart of any project driven by this tool, and in our example project we are now ready to look at this core piece of the puzzle. Essentially, when we think about the infrastructure resources needed to host a site or application like Aegir, there are 3 main categories to think about: Support infrastructure: VPCs and Firewalls, etc. Hosts: usually VMs, sometimes clusters or “bare metal” servers Applications: the software stack from operating system up through the hosted software (eg. Drupal) Our project’s playbooks/ folder mirrors this structure: playbooks/infra - manage support infrastructure playbooks/hosts - manage hosts playbooks/apps - manage applications This setup allows us to separate concerns between plays that are “global” (in the sense they set up the VPC, Firewall, and Project for everything else), those that create a host (on which to run a particular application), and those that configure the software applications themselves. Ansible’s variables system allows us to share important data across these sets of plays as needed. For example, when we create the Firewall we give it a name (web), which is defined in the inventory/group_vars/all/vars.yml in the variable aegir_droplet_firewall. This variable is used when creating the Firewall, but also when creating the Droplet in order to associate the two. To tie things together, we use Drumkit make targets to encode the specific way to call these make targets, and chain them together. We typically break these targets out into smaller pieces in the drumkit/mk.d/*.mk structure that Drumkit provides for exactly this purpose. The early plays in the sequence of spinning up this Ansible-driven infrastructure use localhost as the target host for the playbook (see playbooks/infra/00-up.yml for example). This is because we are making API calls to DigitalOcean from our local workstation, with the aim of creating cloud resources: a Project, VPC, and Firewall, then finally a Droplet. From there, our Aegir playbooks can target the droplet to run on, but in the bootstrapping moment we use localhost Below we’ll break down each of the pieces of the support infrastructure playbooks, but I’ll first note the 2 “wrapper” playbooks to bring all the other pieces up or down: playbooks/infra/00-up.yml playbooks/infra/01-down.yml You’ll note that these two playbooks are nearly identical, but the “down” variation simply sets the “state” related variable (see inventory/host_vars/localhost/vars.yml. Finally, we wrap these up in the Drumkit targets make infra and make infra-down to call these playbooks. DigitalOcean Projects are logical containers within which to house (some kinds of) cloud resources. In our case, it keeps Droplets and Volumes logically separated from others in our DigitalOcean account. Tags are similarly used to group resources with similar attributes. As discussed in Dynamic inventory we use this mechanism to create Ansible host groups. These items are created in the playbooks/infra/project.yml playbook, using the variables we’ve set up to provision the configured project and tags. In our example project, the VPC is a bit of complexity that’s not strictly necessary. As we’ll see, our Droplet will join our Tailscale VPN, which provides an effective means for private networking between Droplets if needed. Obviously we only have a single Droplet in our example, but we could easily scale up to more. That said, VPCs are a typical component of many cloud architectures, and require little extra overhead to provision up-front. In our example project, if we were to add a DigitalOcean managed service into the mix to meet some requirement, we can easily imagine a need for those services (not on our tailnet) to communicate privately with our Droplets. The VPC is created in the playbooks/infra/vpc.yml using a straightforward play that follows the documentation examples, substituting variables for most values. The firewall is the last piece of the support infrastructure puzzle in our example, but of course you can imagine any number of other components and resources coming into play here. By now you should see the pattern and be able to extend it as needed if you have a requirement for a Load Balancer, for example. Add a playbook under playbooks/infra/, following the documentation for the Ansible module related to the resource in question (eg. digital_ocean_load_balancer, and then add it to the list of things in 00-up.yml and 01-down.yml. Introduce variables in appropriate places to make the playbook configurable via your inventory/variables system, and you’re all set. Our firewall setup follows this pattern: playbooks/infra/firewalls.yml follows the digital_ocean_firewall module docs, introducing some basic inbound/outbound rules as appropriate for a webserver. playbooks/infra/00-up.yml One note about the Firewall rules: there is (currently) no Ansible module for modifying firewalls after creation. The DigitalOcean API supports adding/removing rules from a Firewall after creating it, but there’s no access to this feature via Ansible as it stands. Our example project creates the Firewall at the beginning of the process, then our Droplet gets attached to it as we’re creating it. However, what if we needed to add other rules after the Droplet is created, if they needed to encode IP addresses we don’t know beforehand, for example? It would be a straightforward matter to shift firewall creation to happen later in the process. We could simply take the firewall playbook out of the 00-up.yml list, and introduce a new Drumkit target to run later on. In the next post we’ll explore the Ansible Collections we’ve published in the process of building out this infrastructure. In particular, we created a role that will provision a DigitalOcean Droplet in the (opinionated) way we generally want. In future, we may incorporate some of the support infrastructure playbooks discussed above into roles/collections as well. https://consensus.enterprises/blog/pure-ansible-vault/ Wed, 04 Dec 2024 09:00:00 -0500 Variables and Vault on Consensus Enterprises Blog published Wed, 04 Dec 2024 09:00:00 -0500 So far in this series, we’ve covered most of the foundational pieces of our pure ansible infrastructure project. We’ve got Drumkit installed, and configured our dynamic inventory. The last piece we need is a way to configure the fine-grained details of the Ansible playbooks and roles we’ll be running. We also need a secure way to store variables with sensitive data, in such a way that Ansible can access them as needed. The simplicity of Ansible’s approach to these questions is very powerful. The … So far in this series, we’ve covered most of the foundational pieces of our pure ansible infrastructure project. We’ve got Drumkit installed, and configured our dynamic inventory. The last piece we need is a way to configure the fine-grained details of the Ansible playbooks and roles we’ll be running. We also need a secure way to store variables with sensitive data, in such a way that Ansible can access them as needed. The simplicity of Ansible’s approach to these questions is very powerful. The variables system in Ansible integrates with the inventory of hosts and groups, as discussed in the previous post. Variables can be used for just about any value, and are typically represented in YAML files. In our example project, we have 3 key sets of variables: inventory/host_vars/localhost/vars.yml inventory/group_vars/all/vars.yml playbooks/hosts/aegir0.yml The host_vars/localhost variables are very minimal in this example, essentially setting the state of the various infrastructure components, as referenced in the playbooks responsible for bringing up the Project, VPC, and Firewall resources, all of which use localhost as the inventory target to run against. These could be more elaborate if needed, but again: for the simplicity of the project, this is enough. The group_vars/all variables are basically globals, and also fairly minimal. They define things like the name of the infrastructure Project, VPC and Firewall resources. These are used by the infra playbooks as well, but also referenced in the role that creates our Droplet, to associate the resources correctly. Finally, the droplet-specific variables that live in the host playbook are essentially setting the parameters needed by our consensus.infra.droplet role. Here we set the hostname, domain, size, tags, and so on. We also pass credentials to DigitalOcean and Gandi, since the role will need to speak to both of them. Note that the pattern of including the variables directly in the playbook is not best practice, but because the playbook targets localhost, there is no obvious place to create a vars.yml file for it. We could place them in either host_vars/localhost or group_vars/all, but this would break until we have more than one droplet. What we probably need is a structured variable that lists each droplet and this set of parameters (hostname, domain, size, etc) for each. Once again: beyond a handful of hosts, we really need a manifest of some description. Ansible Vault is a purpose-built symmetric encryption tool that integrates seamlessly with Ansible’s native variables system, and makes it possible to store your credentials (think: API tokens, passphrases, etc) within your codebase securely. You can then share a single vault password amongst the team working on the project, and this unlocks everything else they need. This means it’s trivial to store our project-specific secrets and credentials within the project itself. Our example has 2 vault files set up: inventory/group_vars/all/vault.yml inventory/host_vars/localhost/vault.yml As you can see, these files are encrypted, so looking at them directly won’t help. Instead, we need to run ansible-vault to create, view, or edit them. For example: $ ansible-vault view inventory/group_vars/all/vault.yml --- vaulted_tailscale_authkey: "tskey-auth-kjQ4qQ2r4111CNTRL-Z1fYkxcST5HhQCcsKxcm4HS4CGh834jD" vaulted_gandi_access_token: 0c04345ab02f1484d080fd3ae925f2e2a10d80d0 vaulted_outgoing_smtp_password: "some very good password or passphrase to go with the org_technical_contact_email address used for outbound email from this server." vaulted_mysql_root_password: "really_not_a_secure_mysql_password_please_change" The group_vars/all folder contains most of the secrets for this project, and the host_vars/localhost one only the oauth_token for DigitalOcean. Technically these could all live in one place, but the DO token seems more logically associated with our API access to the infrastructure, whereas the other credentials are more from the perspective of inside the infrastructure (they get used on the droplets themselves). We can of course also create other vault files for host or group-specific credentials as needed. Each of these “vaulted” variables will be avaiable to playbooks just the same as regular variables, so we see this pattern frequently in playbooks: - vars: gandi_token: "{{ vaulted_gandi_access_token }}" This will pass the sensitive value to the play or role as the variable gandi_token, which it can use as needed. Of course, all of this hinges on a password for the vault files themselves. We typically use a single password for all the vaults in a given project, but it’s possible to set them individually if needed. We set the vault password when we create it: $ ansible-vault create inventory/group_vars/all/new-vault.yml New Vault password: By default, whenever we run a playbook, Ansible will prompt us for the vault password(s) to unlock those it finds in the project. However, if we configure our ansible.cfg with a simple line like: [defaults] vault_password_file = .vault_pass Then we paste our vault password into a file named .vault_pass in the same place as our ansible.cfg, and it will no longer prompt us. Our example repository even has a handy make .vault_pass target to create this file for us, by simply prompting for the password and writing it to the correct file. In our example project, we have ansible.cfg set to point to an insecure vault password file called dot_vault_pass. If you intend to use this project for your own purposes, you are strongly advised to either remove the two vault files and recreate them with a strong and private vault password, or re-key the existing ones (see notes in the README for details. We have come a long way! Having put Drumkit and our dynamic inventory in place, and now setting up variables and vault files, we are ready to start building the Ansible playbooks that comprise our pure ansible infrastructure project. In the next post, we’ll look at how we’ve structured these playbooks within the project, and then go on to review the playbooks and roles that provision the cloud resources and ultimately the Droplet and Aegir3 software. https://consensus.enterprises/blog/pure-ansible-inventory/ Mon, 02 Dec 2024 09:00:00 -0500 Dynamic inventory on Consensus Enterprises Blog published Mon, 02 Dec 2024 09:00:00 -0500 In any Infrastructure-as-Code project, there is a need to have visibility over what’s actually live in the environment we’re managing. There’s also a need for the Code part to have a manifest for what should exist. Any infrastructure provisioning tool’s job is essentially to reconcile the difference between these two things. In a traditional Ansible setup, these two concepts would be represented by a static inventory (typically a file named hosts). The inventory system within Ansible make … In any Infrastructure-as-Code project, there is a need to have visibility over what’s actually live in the environment we’re managing. There’s also a need for the Code part to have a manifest for what should exist. Any infrastructure provisioning tool’s job is essentially to reconcile the difference between these two things. In a traditional Ansible setup, these two concepts would be represented by a static inventory (typically a file named hosts). The inventory system within Ansible make target systems available for playbooks to apply their plays to. In this sense, a static hosts file is primarily a manifest. The variables system (which we’ll dive into in more depth in the next post) maps onto the inventory of target hosts (and host groups) so that appropriate values (say the size of the aegir0 droplet) can be associated when running plays against those hosts. This is nice because it allows for playbooks that target hosts: all that can enforce (or check) for consistency across all systems in the environment. It gives us the means to assess what’s actually live in the environment, but primarily, the hosts file represents a manifest, or a description of what we expect to be live in the environment. Ansible’s dynamic inventory plugins turn this all on its head. In this context, the inventory is provided dynamically via some script or API (in our case, the DigitalOcean API) to give us a literal and accurate view of what is live. Now, instead of our inventory being built from a static file describing what we want, it’s being generated based on what exists. This is pretty powerful and fairly easy to setup. Following the plugin docs, we create the file inventory/digitalocean.yml, which configures the plugin. Then we configure ansible.cfg to use the plugin. With this setup, we can run an ansible-inventory --graph and see a complete view of the hosts the DigitalOcean is running, in the account for which we’ve provided credentials. We also get some nice features, like a series of variables associated with each host, dynamically populated with attributes like image, id, size, and tags. We can also compose other attributes like public_ip to simplify access to key details about our host. What’s more, we leverage the keyed_groups to dynamically create host groups based on the tags that we assign when we create our droplets. Thus, when we create the aegir0 and specify “web” and “aegir” tags, and then later add a non-aegir webserver web0 tagged only with “web”, we would end up with an inventory that looks like: $ ansible-inventory --graph # Protip: add --vars here to see a full list of vars for each host! @all: |--@ungrouped: | |--localhost |--@web: | |--aegir0 | |--web0 |--@aegir: | |--aegir0 Note that localhost is included above. This target is always implicitly available to Ansible, but we specify a second inventory plugin in our ansible.cfg, which is a traditional hosts file containing only localhost. This lets us associate the ansible_connection=local variable ensuring Ansible doesn’t try to SSH to localhost. We use localhost as a target extensively for plays that make remote API calls, such as creating resources at DigitalOcean. We’ll talk more about the Ansible Vault setup in our project in the next post, but you may notice in our digitalocean.yml file linked above, we specified the oauth_token credentials in a strange way: # This is a trick to extract our oauth_token from our vault before it's fully available to ansible as a "vaulted" variable. oauth_token: '{{ lookup("pipe", "cd `git rev-parse --show-toplevel`; ./inventory/get-token.sh") }}' This was a perplexing part of getting this all set up- the inventory plugin expects us to provide a sensitive value to authenticate itself. Usually, to handle a sensitive value like this, we’d stick this in a vault.yml encrypted variables file, and then use it like so: oauth_token: '{{ vaulted_oauth_token }}' What we discovered, however, is that the vault hasn’t yet been opened at the time the inventory plugin is initializing, and so it gets an empty value and fails to authenticate if we do it the usual way. In fact, the only way we could work out (and kudos to Christopher for this one!) was to wrap a shell script around ansible-vault to “manually” extract the relevant value from the encrypted vault. This get-token.sh script is not complicated, but feels somewhat fragile and out of character in what is otherwise a very robust and thoughtful Ansible ecosystem. Perhaps we’ve overlooked something here, so if you have a better idea to handle this, please get in touch! This dynamic inventory system is very powerful, but there’s a small problem. As described above, we no longer have an inventory that can function as a manifest of what should exist, because it’s always live updating what does exist. In fact, before we’ve created anything, running ansible-inventory would return nothing but localhost! We’ve traded the simplicity of a static file for the accuracy of a dynamically updated inventory. In a large Ansible project, we would turn to an ERP or other directory system that could be a source of truth for “what should exist”, but in a small project, that’s overkill. The good news is, for a small project (1-5 hosts), this isn’t really much of a problem at all. As we’ll see in the way we’ve structured our playbooks in the project, it’s pretty manageable to add a new host by copying one file, editing its variables to suit, and then building an application-level playbook to configure what will run on it. The “manifest” essentially becomes the contents of our playbooks/hosts directory, which is probably good enough for most projects with small footprints. Once you grow beyond a handful of hosts, there probably comes a point where it becomes painful, but a full-blown ERP system still doesn’t make sense. At that point it’s probably worth looking to something like a structured YAML file listing hostnames and a few key parameters (droplet_size, etc) that Ansible can iterate over. For the time being, this method of leveraging the power of dynamic inventory while essentially “hard coding” our hosts manifest in our playbooks feels like an acceptable trade-off, in the name of keeping the system overall as simple as possible (but no simpler). With these pieces in places, we’re ready to add the next piece of the puzzle, setting and associating all the required variables our hosts need for our playbooks to configure them correctly. Watch out for our next post about Ansible variables and vault! https://consensus.enterprises/blog/pure-ansible-drumkit/ Fri, 29 Nov 2024 09:00:00 -0500 Drumkit and other plumbing on Consensus Enterprises Blog published Fri, 29 Nov 2024 09:00:00 -0500 Drumkit is one of my favourite tools that we use at Consensus, because it serves to unify and simplify how we interact with our project tooling. Drumkit itself is very simple, leveraging the venerable GNU make to create short, project-specific “targets” which are essentially a list of sub-commands to run. Any time we have a complex command-line command to do something, and we need to run it regularly, we add a Drumkit target for it and no longer have to remember the correct flags to … Drumkit is one of my favourite tools that we use at Consensus, because it serves to unify and simplify how we interact with our project tooling. Drumkit itself is very simple, leveraging the venerable GNU make to create short, project-specific “targets” which are essentially a list of sub-commands to run. Any time we have a complex command-line command to do something, and we need to run it regularly, we add a Drumkit target for it and no longer have to remember the correct flags to consistently do the same thing. In the case of our example Ansible infrastructure project, we leverage Drumkit for pinning a project-local installation of Ansible (and other tools) at a specific version, but primarily to provide easy access to key operations on the project itself: make tools (drumkit/mk.d/20_infra.mk - ensure we have Ansible, as well as the Galaxy requirements in requirements.yml make infra (drumkit/mk.d/20_infra.mk - run the playbook to create a Project, VPC and Firewall at DigitalOcean make aegir (drumkit/mk.d/30_aegir.mk - combine the 2 following targets to create and configure the Aegir droplet make aegir-droplet - run the playbook to spin up a new droplet make aegir-app - run the playbook to configure Aegir and its dependencies make aegir-down - deregister the server from the VPN and tear down its resources Installing Drumkit is easy enough, and using it is as simple as sourcing the bootstrap script in .mk/scripts/bootstrap.sh. Drumkit installs a symlink named d at the top of your project, so you can simply source d when you first move into the project, and Drumkit will set up your PATH to point to project-local tools, and run any scripts in drumkit/bootstrap.d/. The Drumkit installer creates a number of files and directories you can commit into the repository: .gitmodules & .mk - this is Drumkit itself, git submodule'd into the project. This is also where your project-local binaries go (.mk/.local/bin) d - bootstrap.sh symlink drumkit/bootstrap.d/*.sh - an initial set of simple boostrap shell scripts triggered by source d Makefile - the top-level Makefile, which simply includes Drumkit’s .mk/GNUMakefile, that in turn includes the rest of Drumkit and your project’s make target definitions in drumkit/mk.d/*.mk. drumkit/mk.d/ - the directory where your project-local Makefiles (.mk) will live. I typically create a 10_variables.mk in this file shortly after installing Drumkit to set the versions of tools required. The bootstrap scripts checks for the environment variable DRUMKIT=1 to determine if Drumkit is already bootstrapped in this shell, and then sets that flag if so. It then adjusts your PATH to include the project-local .mk/.local/bin where any tools (like ansible) will be located after you run a make ansible or similar. NB Git submodules are a powerful way to integrate different git repositories, but they do require that you subsequently clone your repository with the --recursive tree every time, in order to also clone the submodules into a new working tree. If you forget this step, you can do git submodule update --init --recursive to fix it :) Ansible is a Python-based tool, and so in addition to the basic Drumkit tooling, we need a basic Python environment within which to operate. Let me say up front, I’m not a very skilled Pythonista. I speak a number of programming languages with varying degress of fluency, and I know enough that I would count Python among them. However, I’ve never been immersed enough in the community and culture of Python to really grok it. I am therefore conscious that I may be missing something entirely in the foregoing section, and hoping some savvy Python pixie will drop me a line to say “here, you can just do it like this” ;) One thing I do understand about the modern Python is that it really wants you to do everything in a virtualenv. PEP 668: Marking Python base environments as externally managed is an example of this trend. For a Python-focused project using Drumkit, this presents a challenge with respect to bootstrapping. We need to activate a python virtualenv (essentially a sub-shell) and have Drumkit manipulate the shell environment. If we bootstrap Drumkit first, then its environment gets clobbered when we activate the virtualenv. However, I couldn’t find a way to “chain” the two together, so I could activate the virtualenv and have it immediately trigger a source d. Even better, I’d love to drop a drumkit/bootstrap.d/00_virtualenv.sh which would do the virtualenv setup as part of a regular source d. What I’ve landed on is using pipenv for its simplicity and ability to manage dependencies and versions effectively. With a simple Pipfile in place to register the python library requirements Ansible will need, we can get started in the project like so: pipenv shell # Activate virtualenv . d # Bootstrap Drumkit make tools # Install ansible and galaxy requirements Note the source d includes a pipenv install from drumkit/bootstrap.d/10_pipenv.sh to actually install the Pipfile requirements into the virtualenv. As a developer, streamlining my workflows and making my daily tools quick and comfortable to use is essential to have the “plumbing” of my project just get out of the way, and let me focus on the problem at hand. With this Drumkit and Python virtualenv setup in place, I have a shared set of versions and tailored-commands for common operations in the project, and I can extend it readily. Having done the bootstrapping steps above, I can run any Ansible tool like ansible-galaxy or ansible-inventory, or equally any of the currently-available Drumkit targets. As I build out more pieces of the project, I implement new Drumkit targets in simple modules under drumkit/mk.d to provide easy “wrappers” around commands. Thus, instead of remembering to type ansible-playbook playbooks/hosts/aegir0.yml && ansible-playbook playbooks/apps/aegir.yml, I can simply run make aegir. We now have the foundation of our Ansible project in place. In the next entry in this series, we’ll build on this tooling to implement Ansible dynamic inventory. https://consensus.enterprises/blog/pure-ansible-simplicity/ Wed, 27 Nov 2024 09:00:00 -0500 Toward infrastructure simplicity on Consensus Enterprises Blog published Wed, 27 Nov 2024 09:00:00 -0500 In the first iteration of Consensus Enterprises’ internal infrastructure, we built a well-provisioned, multi-environment architecture leveraging the most Open services we could find (https://openstack.org). We built things as if we were one of the larger-scale clients we tend to work with, who have legitimate needs for more complex architectures. We implemented a carefully crafted Infrastructure-as-Code (IaC) repository, capable of switching between environments based on shell scripts to set … In the first iteration of Consensus Enterprises’ internal infrastructure, we built a well-provisioned, multi-environment architecture leveraging the most Open services we could find (https://openstack.org). We built things as if we were one of the larger-scale clients we tend to work with, who have legitimate needs for more complex architectures. We implemented a carefully crafted Infrastructure-as-Code (IaC) repository, capable of switching between environments based on shell scripts to set credentials and environment variables. We had shared Terraform state files and thoroughly modularized declarative configuration describing our infrastructure. We had Terraform handing off the configuration to Ansible playbooks so that a single command could build up or tear down the entire environment. We had a lot of (potential) power and a corresponding amount of complexity. In our work with our clients, we often find ourselves advocating for a “keep it simple” approach, only introducing more complexity into a system when it solves a problem we’re currently experiencing. For our purposes within Consensus, we really just need a handful of VMs and some minimal support infrastructure like a VPC and Firewall. Adding more complexity unnecessarily comes at a real cost, both in terms of maintenance fees but also in terms of cloud services bills! Recently we decided it was worth taking some time to rethink our IaC approach to building our own infrastructure, with an aim to simplify as much as possible. There were a few areas of complexity we recognized we could shed. The key piece of technical complexity we identified was the use of 2 different tools to handle our IaC. In principle, Terraform is the “correct” tool for handling the provisioning of Cloud resources, maintaining their state, etc. Ansible is intended as a “configuration management” tool, that’s great for picking up a newly-provisioned Ubuntu VM and turning it into a Webserver tuned for hosting your application. This is great, but some complexity creeps in from the interface between the tools. In order to seamlessly transition from one to the other, you end up with “null_resource” provisioners in your Terraform code to fire off your ansible-playbooks. Those playbooks aren’t quite reusable on their own, because of the context and variables they need which are passed from Terraform. These are manageable considerations but lead to a lot of overhead and churn for what should be relatively simple operations. Ultimately, for the small and simple infrastructure needs we currently have as an organization, using Terraform and Ansible together was simply more horsepower than we needed. In an effort to reduce our costs overall, we recognized that moving to DigitalOcean would save money without any loss of functionality, in terms of what we were actually using with Openstack. What’s more, it’s relatively easy to share access to the DigitalOcean Team, whereas logging into an Openstack Project is.. tedious. We decided to simplify the IaC setup as well, and move to a pure Ansible approach, dropping Terraform entirely. It turns out that Ansible has matured quite a bit when it comes to provisioning infrastructure, and in particular the community.digitalocean collection of modules is very solid and easy to work with. Here again, the Openstack support in both Terraform and Ansible was always a bit uncertain. That said, in principle any cloud provider would work in our setup, especially if their API is well-supported by an existing Ansible collection. Beyond that, we’ve discovered the glory of Tailscale as a dead-simple way to set up a Wireguard-based VPN (and honestly, one of the simplest VPN setups I’ve encountered, period!) We used to provision a Wireguard server ourselves, manage the keys manually, and had a complicated multi-step process to onboard a new team member to login. Now we create a user account, the user logs into Tailscale with it, and they’re done. Finally, we dropped the notion that a single code repository needed to handle deploying the same (or similar) resources into multiple environments. This not only introduced a lot of complexity (conditional logic, variable indirection) in the codebase. It also represented a risk for the developer or operator using the repository, who could easily run destructive operations on the wrong (ahem, production) environment accidentally. We’ve sinced moved to a “dev” IaC repository and a “prod” IaC repository, and they mirror each other in structure and shared libraries (more to come on this), each has config and credentials specific to the environment they target, and are built to focus only on that environment. Having built all this up and deployed it into production over the last few weeks, I wanted to share what we’ve done. I’ve published a couple of Ansible galaxy collections that encapsulate much of what we did, but I wanted to share how it all fits together. To that end, I’ve created an example project in Gitlab called Example Aegir3 Infrastructure that uses these published collections in a way that closely resembles our production setup (simplified somewhat for illustrative purposes). In the remainder of this series, I’ll review how we build a pure Ansible project that leverages our Ansible collections to provision a complete Aegir3 instance on DigitalOcean. https://consensus.enterprises/blog/starshot-moving-drupal-towards-a-product-platform/ Mon, 19 Aug 2024 15:25:00 -0400 Starshot: Moving Drupal Towards a Product Platform on Consensus Enterprises Blog published Mon, 19 Aug 2024 15:25:00 -0400 In the world of Drupal, the terms “product” and “framework” represent two different approaches to how the platform can be used and who it serves. A product in the Drupal sense refers to a ready-to-use solution that allows users to build and manage websites with minimal technical knowledge. It’s about offering a polished, user-friendly experience where the focus is on enabling non-developers — such as content editors or small organizations — to easily create and maintain a web presence without … In the world of Drupal, the terms “product” and “framework” represent two different approaches to how the platform can be used and who it serves. A product in the Drupal sense refers to a ready-to-use solution that allows users to build and manage websites with minimal technical knowledge. It’s about offering a polished, user-friendly experience where the focus is on enabling non-developers — such as content editors or small organizations — to easily create and maintain a web presence without needing to understand the underlying code. The goal is to make Drupal useful and approachable out-of-the-box, with intuitive interfaces and pre-configured options. On the other hand, a framework is more of a toolkit for developers. It provides the building blocks to create highly customized and complex web applications. A framework is designed with flexibility in mind, allowing developers to build everything from basic sites to intricate systems tailored to specific needs. However, this approach often requires a deeper understanding of web development and can be more resource-intensive to implement and manage. Earlier in Drupal’s history, a common refrain was “Kill the Webmaster”. This slogan positioned Drupal as a product, aiming to create an out-of-the-box experience that would lower the barrier for entry for non-technical users. The idea was to incorporate web development best-practices, making it possible for more people and small organizations to publish their own websites without needing to be experts in web technologies. Around the point where Drupal 8 development began, a countervailing trend arose seeking to strip Drupal core down to be more of a framework; known as “small core” at the time. There were multiple reasons for this, including core developer burnout, maintaining such a large codebase. This shift was driven partly by the needs of developers and larger organizations to build complex, custom solutions. While this made Drupal a powerful tool for those with the skills to use it, it also meant that the product-like simplicity envisioned earlier was less prioritized. This evolution left some smaller organizations feeling alienated, as they found it more challenging to use Drupal without technical expertise. The Internet has evolved a lot since the introduction of Drupal 8. But then, so has Drupal itself. Part of the development of Drupal 8 included leveraging the test suite (largely introduced with Drupal 7) to basically rip out and replace a lot of Drupal’s own internals, and make them more object-oriented. This, in turn, was intended to make the upgrade cycle from one major version to the next less painful. It is true that the upgrade process from Drupal 7 to 8 was a very significant lift, but, from Drupal 8 onward, upgrades have been rendered almost trivial! Today, with the recent release of Drupal 11, the vast majority of that movement away from that Drupal 7 world has occurred within Drupal itself, but a significant number of Drupal sites are still on Drupal 7. In parallel, we’ve seen an explosion in the popularity of comparable projects like Wordpress, which takes that product approach and runs with it. Now that Drupal core has evolved into the powerful framework it is today, the community is no longer faced with an either-or choice. We can, in fact, have our cake (Drupal core) and eat it (Starshot) too. Recently, Consensus colleagues Dan Friedman and Brian Sharpe had the chance to talk at DrupalCamp Ottawa about the promise of Starshot. Starshot represents a shift in the Framework-vs-Product conversation, bringing the best of Drupal’s framework into the product development arena. We see this as an opportunity for Drupal to say to those smaller organizations: “We’re going to build something that works for you, for the skillset that you actually have.” In short, “Kill the Webmaster” has become “Low Code/No Code”. We think this turn toward simplicity and product focus provides some very exciting opportunities. Automatic Updates will provide an upgrade process that doesn’t require an entire devops team to keep running. Behind the scenes, the Drupal infrastructure team is integrating software supply chain security into their packaging pipeline. The Drupal Association has sponsored the development of a Composer plugin and Rugged, the server-side component developed primarily by our colleague, Christopher Gervais. As a result, out-of-the-box Starshot (and Drupal 11) will provide users the confidence that the software they have requested is in fact the software they are receiving. Project Browser will allow users to manage installed modules and themes. Because this involves downloading new code into a running site, it has the potential to be insecure. Here at Consensus, we’re also developing Aegir 5, a self-hosting platform in which it will be relatively simple for the user to temporarily grant access for this task, then re-secure the site, from outside of Drupal itself. Other hosting providers will likely follow suit. Drupal core will be free to streamline the “framework” even further. Its modular approach allows us to selectively enable only the components we need, like choosing not to use Node when it’s unnecessary. This gives us the flexibility to build custom tools and configurations without relying on every part of the traditional Drupal components, tailoring the CMS precisely to our needs. Most of the previous items are “behind the scenes:” folks responsible for maintaining a site wouldn’t normally interact with most of them on a regular basis. Where things start to get really interesting is in seeing Drupal itself as a product development platform. Typically, a Drupal site is made up of: Drupal Core, providing base functionality; Community contributed modules, providing additional functionality; and, Themes, granting control over the site’s look and feel. Install profiles, which do all of the initial setup, get us to the point where we have a site that “does a particular thing” – they have always been the thing that we think of at Consensus when we think of a Drupal site being a particular application. But, there are a set of limitations on install profiles that have, historically, held Drupal back from being a product-oriented platform. For example, crucially, once you have installed a site using a certain profile, you can’t shift to a new one (via profiles alone – you need to involve migrations, or other mechanisms). So, if you offer a service that has Bronze, Silver and Gold packages, you can’t move a given client site from one “level” to the next via install profiles alone. They don’t, by themselves, provide a long-term maintenance or sequential migration strategy for any given web site “product.” In the course of various development projects at Consensus, we’ve had to come up with strategies for automating import/creation of various classes of data, including demo content, default content, test content, and so on. So we’re excited by Recipes that provide simplified content import, among other “low-code” (YAML only) interventions. This provides possibilities for getting past the limitations of install profiles. Another interesting inclusion in Starshot is the notion of a single-directory application, that further integrates 3rd party tools (javascript libraries, and so on) into a Drupal application, with minimal developer intervention. All in all, we see Starshot as a tremendously exciting and fascinating next step in Drupal’s evolution toward a product development platform. We look forward to supporting it, both through our ongoing work with the Drupal Association, and our continued work on Aegir 5. Drupal has evolved into a powerful platform for developing digital products, with tools like Starshot enhancing its versatility. Whether you need a straightforward website or a more complex application, Drupal’s open-source nature and flexibility make it a great option. At Consensus Enterprises, we’re passionate about open-source technology and committed to helping you achieve your goals. If you’re looking for a partner who values innovation and practicality, let’s talk about how we can use Drupal to make your vision a reality. Reach out to us today to get started. https://consensus.enterprises/blog/drupal-10-on-aegir-3-a-step-by-step-guide/ Mon, 24 Jun 2024 09:00:00 -0400 Drupal 10 on Aegir 3: A Step-by-Step Guide on Consensus Enterprises Blog published Mon, 24 Jun 2024 09:00:00 -0400 For those in a hurry: Start with a fresh Ubuntu 22 VM. Clone this git repository and follow the instructions in the README file. Use the Aegir 3 site migration process to move your Drupal sites to your new setup. Happy migrating! For more detailed information, keep reading. At Consensus Enterprises, we’re dedicated to helping organizations transition smoothly from Drupal 7 to Drupal 10 on Aegir 3. While we’re developing the future of Drupal self-hosting with Aegir5, we continue to see a range … For those in a hurry: Start with a fresh Ubuntu 22 VM. Clone this git repository and follow the instructions in the README file. Use the Aegir 3 site migration process to move your Drupal sites to your new setup. Happy migrating! For more detailed information, keep reading. At Consensus Enterprises, we’re dedicated to helping organizations transition smoothly from Drupal 7 to Drupal 10 on Aegir 3. While we’re developing the future of Drupal self-hosting with Aegir5, we continue to see a range of users in the Aegir community, from small organizations through academic institutions to large government departments, who need to keep their existing Aegir 3 installations working. In particular, as Drupal development moves forward, the requirements of Aegir 3 itself and modern versions of Drupal core have begun to drift apart. Of course, the medium-to-long term solution we recommend will be to migrate to Aegir5, but until it becomes available, the community needs a way forward. As one of Consensus’s DevOps specialists, I started documenting how Aegir 3 could continue to host modern Drupal versions when it became clear that the old Aegir 3 Debian repositories were no longer sufficient. This led me to create an example repository that anyone can use to deploy and configure an Aegir 3 instance for hosting modern Drupal versions on Ubuntu LTS in about 15 minutes, utilizing our Consensus Ansible roles. When long-time Aegir and Drush contributor Steven Jones started working on a solution for hosting Drupal 10 on Aegir 3, I was particularly keen to see what he would come up with. As Steven writes in his 2-part series on the subject, there are several tricky interactions between Drupal core, Drush, and Aegir’s Provision component that require tending to, in order to make the whole stack work together. He also points out the crucial caveat that this solution should be regarded as temporary, since it is brittle, and has to dance around conflicts between Drush 8 and Symfony 6. His articles are great, and I’m not going to rehash his eloquent explanation here. However, he does not quite fill in all the details one needs in order to achieve a running Aegir instance that supports Drupal 10. My goal was, thus, to extend my earlier work and provide a working example that Aegir users can then customize to meet their local needs. Building on Steven’s recipe, we need several ingredients to build an Aegir 3 that can host Drupal 10: Use BOA Drush 8 Add Drupal 10 Support to Provision Address Drupal Core Asset Handling Changes Resolve Conflicts Between Symfony 6 and Drupal 10 We accomplish the first of these via our Ansible playbook and our Ansible roles; the second and third via patches to Provision; and the fourth via a custom Drupal 10 Aegir 3 Platform. All the code for these solutions is public and freely available under open-source license terms, and we are actively working to contribute the Provision patches upstream. Drush 9+ fundamentally refuses to be anything other than site-local; and, for better or for worse, Aegir 3 fundamentally relies on global Drush functionality, so we must stick with Drush 8. Aegir5 will solve this problem by adopting more flexible DevOps tools like Ansible; but, for now, we follow Steven’s suggestion and use BOA’s version of Drush 8 in our Ansible playbook. Provision doesn’t include support for Drupal 10 (yet)–but we can add it from BOA Provision, as Steven suggests. This causes a few minor problems, because the files from BOA that support Drupal 10 assume they are living in the BOA ecosystem, which breaks vanilla Provision. With some careful testing, we’ve found and removed those assumptions, and thus fixed the problems. Meanwhile, Drupal core has changed the way it handles CSS and JavaScript assets, which requires fixes in Provision’s Nginx config generation code. Thanks to Consensus colleague Scott Zhu Reeves for help tracking these down so we can ensure Provision generates the correct Nginx config! As Steven says, “Drupal 10 uses Symfony 6, which has type hints on various interfaces and methods, and Drush 8 uses Symfony 2, which does not. So, if you load one before the other, then as soon as PHP tries to load the second, it’ll die because either the types are there, or they aren’t.” He walks through the steps required to resolve this dilemma, which we won’t repeat here; long story short, we are following his suggestions. We do it all with Composer, as you can see via these links to our open-source Drupal 10.2 Aegir Platform: Remove typehints from Drupal core Downgrade psr/log Patch symfony/console By following the steps at the top of this post, you can bridge the gap from Aegir 3 to Aegir5, ensuring continuity and modernizing your Drupal hosting environment. If you need assistance or have questions, our team at Consensus Enterprises is here to help. Let’s navigate the migration journey together and keep your projects running smoothly. For detailed instructions and support, feel free to contact us and follow us on LinkedIn. https://consensus.enterprises/blog/computed-token-field-intro/ Sat, 20 Apr 2024 09:00:00 -0400 Introducing Computed Token Field on Consensus Enterprises Blog published Sat, 20 Apr 2024 09:00:00 -0400 In a recent project for a federal government client, we needed to upgrade (rebuild, improve, and migrate content) an internal application from Drupal 7 to Drupal 10. One of the challenges we faced was to build reports that display a large set of data fields pulled from a complex data model. We found that using Views alone was insufficient to meet the performance and maintenance requirements for these reports. This led us to develop Computed Token Field as a way to streamline Views configuration … In a recent project for a federal government client, we needed to upgrade (rebuild, improve, and migrate content) an internal application from Drupal 7 to Drupal 10. One of the challenges we faced was to build reports that display a large set of data fields pulled from a complex data model. We found that using Views alone was insufficient to meet the performance and maintenance requirements for these reports. This led us to develop Computed Token Field as a way to streamline Views configuration required to produce the necessary reports. The Computed Token Field has provided us with several key benefits: Performance Boost: Reducing the number of relationships in Views has led to faster and more efficient database queries. Maintenance Ease: With a simpler Views config, the system is easier to manage. Scalability: Allowed for managing a large volume of data with minimal added complexity or degraded performance. Using Drupal’s Token system, the Computed Token Field module dynamically populates fields based on predefined tokens. This capability enables site builders to easily implement and adjust computed data fields, utilizing Drupal’s strong caching to ensure data consistency. This client’s application manages the complex review processes for all publications the department produces. The system holds data about the publications being produced, rather than the publications themselves. This metadata nonetheless represents a large volume of information collected while reviewing publications. Each stage of the workflow process is handled by a different user or role, and securing access to the data entered in other stages was critical. To address this, we developed a hierarchical data model using ECK entities to create boundaries around sets of data fields that needed to be treated distinctly in terms of access control and workflow state. The hierarchical data structure had several levels of content entities: Node types (Series): Top-level “publication” nodes, representing 4 different types or “series” of publications. Publication Parts: Content entities linked by Entity Reference fields from the node, to contain data collected at different workflow stages. Sub-Parts: Further bundles within the Publication Part ECK entity type, but linked from the main Part rather than the node. Each of these was a distinct content entity, linked by Entity Reference fields. Certain fields on these entities were shared across all Publication Series, others were shared but appeared on different Parts, and others were unique to a particular Series. From the perspective of access control, this structure served us very well. However, another key function of this application was to provide comprehensive reporting and data export features, driven by the venerable Views. The reporting Views needed to display a large set of fields across any of the publication Series, and provide a comprehensive set of filters to narrow down the results by certain criteria. Using standard Views configuration, we would need to add a Relationship for each of a large set of entity data tables. This introduces a great deal of complexity into the SQL query, as well as the configuration of the View itself. The immediate concern we saw was one of performance. Even with the relatively moderate tens of thousands of publication records in the database, the query produced by introducing so many relationships was starting to become cumbersome. Equally important, maintaining such a complex View configuration was a significant concern, primarily because this client needed to be able to improve and tweak this reporting aspect of the site on an ongoing basis. We were able to reduce some of the complexity by sharing the Publication Part reference fields wherever possible, thus reducing the overall number of relationships required. This simplification helped a lot, especially when there was overlap in fields. However, in cases where the same field (eg. ISBN) existed on different parts on each Series, for example, we’d need a relationship for each one, add the field/filter for each relationship, and then collapse them to appear as a single field in the Report. This was all viable in principle, but seemed untenable in practice. With the idea of limiting the number of Relationships in our View configuration, we took a step back to consider alternate approaches to this problem. One idea was to use Computed Field to “mirror” some of these tricky fields on the Node entity. If we treat these as a kind of “local cache” of key field values, they become easier to add directly as Views fields and filters. This technique turned out to work quite nicely. We recognized some risk with keeping data in sync, inherent whenever data is duplicated in a system. However, our initial prototyping indicated the Computed Field’s feature set would ensure the computed field was always updated appropriately any time the “source” field changed. However, there was one challenge with using Computed Field as-is: there is no UI beyond “write a PHP function”. Not only would it become tedious to write a function for even a handful of fields, but this bar was too high to expect our client to maintain and extend. We needed to give staff an interface, to be able to configure the “compute function” through Drupal’s admin interface. This is where Token module came into play. We realized that our customized Computed Fields could take a configurable Token associated with the field, to pull the value from the source field and copy it up to a field on the node. This makes it trivial to add to a View without the need for an extra relationship. Thus, we could write a single function to simply render the token value provided by the field configuration, and use that to populate the value of the field. For example, we had a Computed Token Field on the “Formal Publication” series node type, configured with a token value like [formal_publication:field_part2:entity:field_part2c:entity:field_actual_release_date:value]. This computed field behaves like a Date field, and could thus be included as both a field in the Report and a filter to narrow down results. Another key feature we needed from these “cache” fields was that they “look and feel” like fields of the same type as their source. We identified 3 key types of fields we needed to support: Date fields, Text fields, and Entity Reference fields. We set out to extend the “core” Computed Field implementations of FieldType plugins of these types, but only String (short and long) fields were supported by Computed Field. The Entity Reference and Date field implementations in Computed Token Field required some delving into the core FieldType plugins for these types, as well as the details of how Computed Field manages the various FieldTypes it does support. In the end, these classes ended up rather clean, extending the core FieldType classes but incorporating the ComputedFieldItemTrait to make them act like a computed field. The majority of the code for these classes (even the String ones) lives in a trait that provides the field settings form as well as the executeCode() method, which does the actual rendering of the token. Notably we don’t support multi-value scenarios well, and would welcome feedback and ideas on the d.o issue. Having established this functionality and improved the configuration and performance of our reporting Views, we noticed another use-case for Computed Token Field. The publication management application sends emails to key stakeholders as a given publication moves through different stages of its workflow. Who gets an email is determined by the value in a user reference field somewhere in the data about that publication. We were using ECA to accomplish this, setting up Events based on workflow state changes, and Actions to send out emails. The content of the emails was shared across Series, but the field pointing to the user to email was on different Parts for different Series. This meant that we’d have duplicate ECA rules for different Series whose only difference was which field on which Part we looked up the user email. Once again, the technique of “caching” a copy of the user reference at the node level allowed us to streamline the number of ECA rules required. By moving the value from the various Parts into a shared field on the Node, we could configure our ECA rules to look for the relevant email address in one place, regardless of which Series it was. We see potential for the Computed Token Field module to address advanced data management needs, particularly in sectors like government, education, and healthcare where complex data structures are common. However, we’re interested in hearing from the Drupal community about other possible applications. If you have thoughts on how the Computed Token Field might be integrated into your projects, or suggestions for further enhancements, we’d like to hear from you. Your feedback can help us refine this tool and explore new applications within the Drupal ecosystem. We look forward to discovering how you might leverage the Computed Token Field to streamline your data management tasks! https://consensus.enterprises/blog/aegir5-stewardship/ Mon, 23 Oct 2023 08:00:00 -0400 Aegir5 Stewardship: Releasing Aegir back into the wild on Consensus Enterprises Blog published Mon, 23 Oct 2023 08:00:00 -0400 We are pleased to announce we’ve posted the first Aegir Dispatch on Aegir5 to relaunch the community site! As mentioned in the post, we have continued evolving our list of high-level User Stories describing the critical functionality required to plan a series of Releases. We are continuing to break down these stories into a series of Request For Comment (RFC) Release Epics. This is work in progress, but we are eager to hear any feedback you may have :) Our goal is to understand the relative … We are pleased to announce we’ve posted the first Aegir Dispatch on Aegir5 to relaunch the community site! As mentioned in the post, we have continued evolving our list of high-level User Stories describing the critical functionality required to plan a series of Releases. We are continuing to break down these stories into a series of Request For Comment (RFC) Release Epics. This is work in progress, but we are eager to hear any feedback you may have :) Our goal is to understand the relative importance of Aegir 5 features in the broader community. This can inform our priorities as we formulate a Release plan. Aegir belongs to its community, but until recently, we haven’t been in a good position to accept help on Aegir5 for a variety of reasons. We are actively trying to re-establish a clear boundary between our roles as Consensus workers and Aegir project participants and maintainers. In addition to planning and estimating, our immediate focus has been on unlocking developer contributors, to allow broader participation in the current push to build out the initial functionality. To that end, we’ve been cleaning up our documentation, marking older issues as Deprecated, and other housekeeping tasks to make it easier to grok where things are at and where to engage. We are actively reaching out to anyone who’s been in touch about this, as well as those we know have previously had interest in the project. By Release 3 we expect things to be stable enough for “site builders” to experiment with moving sites onto Aegir 5. In the meantime, we are working in the open and soliciting input and participation in defining and prioritizing features. For Consensus, this is a passion project that also underpins our business strategy well into the future. We intend to pursue building out the key use-cases we can see have the most value for ourselves and the community. We encourage and welcome other contributions of course, but these are the things we’re focused on: Estimating the User Stories described in the series of Releases we’ve laid out. Translating Story Points into a level of effort guess, and ultimately a development budget. Secure funding and dive into implementing the Features described. If you have questions, comments, or want to get involved, feel free to drop into our public Matrix channel and say hi! https://consensus.enterprises/blog/aegir5-roadmap-update/ Tue, 03 Oct 2023 08:00:00 -0400 Aegir5 Roadmap Update: Planning & Estimation on Consensus Enterprises Blog published Tue, 03 Oct 2023 08:00:00 -0400 Back in June, we submitted a Pitch-burgh pitch to seek funding to finally get Aegir 5 fully off the ground. Prior to that, we had been working to build a roadmap to push our prototype implementation up to feature parity and an initial release for broader community participation. In parallel to that, we’d been publishing this series of posts with the goal of culminating in a roadmap and action plan. Unfortunately, we didn’t get selected at Pitch-burgh, and then summer vacation came, things got … Back in June, we submitted a Pitch-burgh pitch to seek funding to finally get Aegir 5 fully off the ground. Prior to that, we had been working to build a roadmap to push our prototype implementation up to feature parity and an initial release for broader community participation. In parallel to that, we’d been publishing this series of posts with the goal of culminating in a roadmap and action plan. Unfortunately, we didn’t get selected at Pitch-burgh, and then summer vacation came, things got busy, and the project slid to the backburner (again). Fortunately, our efforts to be more public about our work on this project seem to have attracted some attention. We’ve already heard from a number of folks who are interested in using Aegir 5, wanting to know when it’ll be ready, and offering to help in various ways. We’re very grateful to have several offers of help with development/testing or funding. However, we haven’t been in a good position to accept support, as we haven’t documented what needed doing in enough detail, nor how much it would cost. We still needed a roadmap, and we recognized that our previous work on this had been overly focused on technical details and not enough on user-facing value. With that in mind, we’ve initiated a new planning & estimation mini-project, to describe and estimate a core set of User Stories for Aegir 5. In a couple weeks, we will be sharing the outcomes of this process publicly, and seeking feedback (RFC style) in our gitlab issues. This feedback will help to understand the importance and value of the variety of features Aegir 5 might need from a broader perspective. We can use this to prioritize groups of User Stories into a Release plan, and thus derive an overall estimate of effort and cost. This should better enable us to receive help, either by funding development or incorporating outside contributions. We aim to (re)build a developer community and Aegir 5 ecosystem, and to empower the formidable Drupal community to leverage this toolkit to its own ends. Along with the offers of support, we’ve also heard a variety of concerns about the future of the Aegir system or Aegir 5 in particular. For example: Can Aegir 3 support Drupal 10? Will there be a migration path from Aegir 3 to Aegir 5? Does Aegir 5 support multi-site environments? Can Aegir 5 satisfy a SaaS use-case? The first 2 of these we can answer directly: No. Fundamental divergence between PHP8, D10, and Drush make this effectively impossible. Yes! Onboarding Aegir 3 sites to Aegir 5 is a key use-case on our radar. Questions 3 and 4 warrant more detailed answers, which we plan to post about shortly, but the short answers are “sort of” and “most definitely” respectively. The Aegir 5 architecture in general is a lot more flexible, intended to enable use-case-specific pluggability. We recognize a growing sense of urgency for Aegir 3 users facing the imminent D7 EOL to find a new home, and we’ve set aside some time for our team to spearhead a collective effort to make Aegir 5 a viable solution. Our plan is to concentrate on enabling contributions to that collective effort by a variety of means (these blog posts, improved docs, transparent dev workflows, etc). Stay tuned here for upcoming posts mentioned above, as well as announcements and calls for feedback. We expect to be looking for input on a first set of User Stories in a week or two, and again when we publish a release plan shortly thereafter. If you have questions, comments, or want to get involved, feel free to drop into our public Matrix channel and say hi! https://consensus.enterprises/blog/aegir5-building-the-future/ Tue, 27 Jun 2023 09:00:00 -0400 Building the Future of Drupal Hosting with Aegir5 on Consensus Enterprises Blog published Tue, 27 Jun 2023 09:00:00 -0400 Consensus Enterprises is not your average technology company. We are a collective of workers who are driven by optimism, equity, and a shared passion for open-source technologies. Our mission is to build a company that doesn’t just benefit a select few, but instead shares the rewards of our collective hard work. Having contributed to the Drupal community and others for over 20 years, open-source technology is deeply ingrained in our DNA. We maintain the widely-used Aegir hosting system, a … Consensus Enterprises is not your average technology company. We are a collective of workers who are driven by optimism, equity, and a shared passion for open-source technologies. Our mission is to build a company that doesn’t just benefit a select few, but instead shares the rewards of our collective hard work. Having contributed to the Drupal community and others for over 20 years, open-source technology is deeply ingrained in our DNA. We maintain the widely-used Aegir hosting system, a trusted application manager for countless Drupal sites. Through our efforts, we’ve delivered numerous modules, patches, bugfixes, updates, and talks to the community. The Drupal community has long enjoyed the benefits of a robust, open-source tool in the form of Aegir for the secure and efficient management of their web applications. However, the impending End-Of-Life (EOL) for Drupal 7, which underpins the current version of Aegir, is casting a shadow on this reliable tool. With the EOL, Aegir3 will also sunset, leaving a gap in the market for a community-owned, open-source hosting solution. This situation presents us with an urgent need to create a modern alternative that is not only reliable but also doesn’t require one to be a DevOps expert to operate. The community needs a tool that can support the transition to Drupal 10 sites, while also accommodating non-Drupal sites. Meet Aegir5, our proposed solution to this pressing issue. Envisioned as an advanced hosting solution, Aegir5 is more than just a replacement for its predecessor. It is a tool designed for the future, levering modern components and offering flexibility for a variety of use-cases. Being application-agnostic, Aegir5 will not only offer a smooth migration path for Aegir3 users but will also extend its benefits to Drupal developers, development shops, small non-profits, large institutions, Software-as-a-Service developers, and Platform-as-a-Service providers. Aegir5 will be poised to manage entire software stacks, including infrastructure, marking a significant evolution in the Aegir hosting platform. Achieving this vision for Aegir5 requires both time and resources. We already have a solid proof-of-concept, but we need to solidify and stabilize it. We estimate around 400 hours of effort to reach the Minimum Viable Product (MVP) stage for Aegir5. That work will include the creation of a functional Drupal application hosting platform and a stable developer-ready toolkit, comprehensive documentation and a test suite. And it will allow for contributions from the wider community. We recently shot a short video pitch that was included in the DrupalCon Pitch-Burg pitch competition. Bridging to the Future: A plan to bring Aegir5 to the Drupal community from Consensus Enterprise s on Vimeo. Description: Consensus Enterprises is seeking funds to match its own investment in building Aegir5: an application-agnostic, modern, and open-source community-owned hosting tool that works with Drupal 10. This will offer a migration path for Aegir3 users, and will also enable management of non-Drupal sites and applications with entire software stacks. This funding will help us reach an alpha MVP more quickly, opening the door for other contributors in the community to also start developing the application. The successful development and launch of Aegir5 will mark a major milestone for Consensus and the Drupal community at large. Not only will it solve the imminent hosting problem, but it will also pave the way for future development and innovation within the Drupal environment. Moreover, it will continue our tradition of offering a robust, community-owned, open-source alternative to commercial hosting products and services. Aegir5 is more than just a product; it’s a testament to the power of collective hard work, embodying our commitment to sharing the rewards of such work with the community. We invite you to be part of this exciting journey. Once we release the Alpha, we will be able to open up development and contributions to the bigger Drupal community. If you are an open-source enthusiast, a small business, a non-profit organization, a university or government agency or an organization simply looking to receive support from a collective of highly-experienced, strategic and kind technologists, then let’s explore how we can help you solve your technical problems. Get in touch today and learn how you can support or benefit from the expertise of some of Aegir5’s architects. Stay tuned for more as we continue to share updates on the development of Aegir5. https://consensus.enterprises/blog/aegir5-feature-parity/ Tue, 06 Jun 2023 09:00:00 -0400 Aegir5: Feature parity between Aegir3 and Aegir5 on Consensus Enterprises Blog published Tue, 06 Jun 2023 09:00:00 -0400 In previous posts, we’ve covered our Kubernetes framework for an alternative back-end to Aegir5, as well as the front-end Tasks and Operations and Clusters, Projects, Releases, and Environments. We also discussed the Queue architecture that ties the front- and back-ends together. This time, let’s consider the planned feature parity between Aegir 3 and Aegir 5. In Aegir3, Servers only have a “verify” task. This ensures that the front end can connect to the server over SSH. The new model will … In previous posts, we’ve covered our Kubernetes framework for an alternative back-end to Aegir5, as well as the front-end Tasks and Operations and Clusters, Projects, Releases, and Environments. We also discussed the Queue architecture that ties the front- and back-ends together. This time, let’s consider the planned feature parity between Aegir 3 and Aegir 5. In Aegir3, Servers only have a “verify” task. This ensures that the front end can connect to the server over SSH. The new model will introduce the analogous Cluster entity. This will work similarly, ensuring that the front-end can connect to the Kubernetes cluster. Platforms, for their part, generally have two tasks in Aegir 3: verify, and migrate sites. The Verify task largely is responsible for deploying the platform, as well as ensuring proper file permissions and so forth. Aegir5 represents a notable departure in this area. We’ve introduced two different entities to represent different aspects of what a Platform does in Aegir3. First, we have Projects, which are effectively a codebase- a git repository somewhere with an application that’s under development, evolving and improving. On top of that is a Release, which in the Kubernetes backend amounts to building a new container image. This represents a snapshot of the Project at a specific point in time: a version of the application, all its dependencies, and the runtime environment config needed to run it. In the short-term, our focus is on this cloud-native use-case, but the idea of Projects and Releases is applicable even in a more traditional VM-based scenario. At the infrastructure level, Clusters is the current target, but Aegir will support a more traditional Server as well, allowing Projects to be deployed onto varying infrastructure models. In Aegir3, Sites have a CRUD suite of tasks that include installing, verifying, upgrading and deleting. In Aegir5, this will be represented by Environments, into which we can deploy, update and delete the application. From the front end, this will look almost identical, but of course in the Kubernetes context these represent a running instance of the application: a container image of a particular Release, on a given Cluster. There is one more important distinction at this level. Aegir3 relies extensively on Drupal’s multisite capabilities. With Aegir5, we are setting aside that functionality almost entirely. Instead, we treat each site as standalone. This allows for support of large sites with complex development workflows. However, we can also track which sites use a given image. As a result we can (with a single operation) trigger migrations/updates for all the instances that are using that image. Aegir3 relies heavily on a bespoke task queue. As described previously, these Tasks have become Operations in Aegir5. They already have a log of the backend output. Operations in turn are composed of Tasks, which in Aegir5 are more granular, configurable steps in a larger whole. For example, writing a virtual host configuration, or setting up a HTTPS certificate. As discussed, we’ve modernized the task queue itself to use Celery, which is built atop RabbitMQ. This stack replaces the bespoke implementation and unlocks a more distributed “control plane” for the Aegir system. In addition to the task queue, Aegir3 also includes queues for running cron, taking scheduled backups, and so forth. Celery is certainly capable of implementing such queues. However, these are not an immediate priority. One feature from Aegir3 that we would like to incorporate is the ability to retry failed tasks. However, this may not make sense in a Kubernetes context. Backup and restore are important tasks for Aegir. In the long run, Aegir5’s Kubernetes backend will probably want to do volume snapshots using a tool like Valero. However, to keep things simple to start, we will operate similarly to how we do this in Aegir3, using an SQL dump and file tarball, a model which has served us well all these years. Cloning a site, in the long run, will be a composition of deploying from a snapshot/backup, taken immediately. Here again, the composition of this kind of operation becomes very similar to Aegir3. However, we can also simply deploy a new site, and then synchronize the database and files from the source site. Disabling a site will be a matter of running a job that rewrites the vhost to point to a static page, similarly to how Aegir3 does it now. As previously discussed in the Frontend Low-level Architecture post, the password reset functionality is much improved, by simply providing a “login to site” button. URL aliases and HTTPS capabilities are both already handled within the Kubernetes backend via the ingress and certificate manager services. These will be exposed via the Drupal UI as configuration options when creating an Environment to deploy a Release. The fundamental concepts of Aegir3 map quite readily onto the new Aegir5 architecture. There are some functional changes to modernize the overall framework, but from an end-user standpoint, the user experience (UX) of working with an Aegir5 system should feel quite familiar. The mapping described here shows how the new back-end and queue components integrate into this UX structure. In our next post, we’ll lay out our plan to move ahead and implement the missing pieces to get a fully functional MVP Aegir5 system. We are excited to be talking about this effort more publicly, and even more so to see more engagement with the Aegir5 project from the wider community. Let us know what you think and how you want to get involved! https://consensus.enterprises/blog/aegir5-pitchburgh/ Wed, 31 May 2023 09:00:00 -0300 Aegir5 + Pitch-burgh on Consensus Enterprises Blog published Wed, 31 May 2023 09:00:00 -0300 See our recent Aegir5 Roadmapping series for more background! Recently, Dries announced an exciting event at the upcoming DrupalCon Pittsburgh: Shark Tank meets Drupal: pitch your best innovation ideas/): Announcing “Pitch-burgh”, an innovation contest at DrupalCon Pittsburgh, where members of the Drupal community can pitch their ideas to receive funding. […] The entrepreneurs give short 2-3 minute presentations in the hopes of securing funding for their idea. Since we’re actively pursuing … See our recent Aegir5 Roadmapping series for more background! Recently, Dries announced an exciting event at the upcoming DrupalCon Pittsburgh: Shark Tank meets Drupal: pitch your best innovation ideas/): Announcing “Pitch-burgh”, an innovation contest at DrupalCon Pittsburgh, where members of the Drupal community can pitch their ideas to receive funding. […] The entrepreneurs give short 2-3 minute presentations in the hopes of securing funding for their idea. Since we’re actively pursuing partners and funding for Aegir5 development, this seemed like it could be a good opportunity. If nothing else, it’d give us a chance to develop a succinct message to describe this initiative. We’re pretty happy with the results. We hope you like it too (and read on below for more about the making of, and intention behind our pitch): Bridging to the Future: A plan to bring Aegir 5 to the Drupal community from Consensus Enterprises on Vimeo. Description: Consensus Enterprises is seeking funds to match its own investment in building Aegir 5: an application-agnostic, modern, and open-source community-owned hosting tool that works with Drupal 10. This will offer a migration path for Aegir 3 users, and will also enable management of non-Drupal sites and applications with entire software stacks. This funding will help us reach an alpha MVP more quickly, opening the door for other contributors in the community to also start developing the application. We had recently started working with an absolutely fantastic marketing consultant, Zack (storypanda.ca). When we asked him about it, he was quick to recommend Rob, with whom he’d collaborated successfully on previous projects. Rob (robspence.tv) is an amazing filmmaker and, interestingly, also a real-life cyborg. They both quickly agreed to help, so we started figuring out what we wanted to say. Zack led some initial brainstorming, and gave us some homework in the form of a questionnaire to help the Consensus-Aegir team to articulate our vision for the project. In the meantime, Rob had us scout locations within our respective homes. He also prompted us to start gathering some visual elements to incorporate into the video, so that we wouldn’t just be talking heads. Zack came up with a draft script that touched on all the important topics we’d come up with. After a round of revisions, we were ready to shoot. None of us felt particularly comfortable in front of the cameras, even if they were just the ones in our phones. Rob came through again, putting us at ease with witty banter, and confidence earned over decades of shooting and editing videos. With the shooting done, Rob put together a rough-cut. This was eye-opening, as we could finally get a sense of what the final product would look like. Several edits later, based on feedback from the team and Zack, we had our final product. To be frank, we’ve found it challenging to distill a succinct message in terms of our vision of Aegir, what it can be for the community, and what we want to do with it. “Here’s what we’re proposing…” Christopher begins, about 30 seconds into the video. He goes on to describe the urgent need to sunset Aegir3 and replace it with Aegir5. We’ve been cultivating this for several years, but we recognize the need to re-open the Aegir project to community participation and contribution. In the video, we identify a few deliverables, which focus on releasing Aegir5 back into the Drupal community at large. The core of this milestone is to wire up the functional components we’ve built into a coherent minimal viable product (MVP). This will address the immediate use-case we’re seeing in our own work, while allowing the flexibility to expand functionality to other cases. Fundamentally, the MVP will provide a functional Drupal app hosting platform which can replace Aegir3, ideally before Drupal 7 goes EOL. In the Alpha MVP stage, the goal is to release a stable “core” of Aegir5, a developer-ready foundation on which to build further. Improving the developer experience (DX) in Aegir5 is a key design goal, since we want the system to support a wide array of features we can’t build all on our own. To that end, we plan to review, update, and improve our documentation, continue building out example modules which demonstrate how to leverage the underlying tools, as well as a comprehensive test suite. We view testing as both a quality control, guarding against regressions and other breakage, but also a means to illustrate how the system behaves. We’re working in the open on this, so feel free to check out the milestone details and engage with us on the GitLab project anytime :) Once we complete the Alpha MVP milestone, we will start incorporating feedback from the existing Aegir developer community (shout-outs to Omega8.cc, DevShop, and InitFour.nl!) and beyond. We’ve already had some great conversations with existing Aegir3 users, and we plan to evolve the Beta MVP (and subsequent) milestone(s) based on the interest and feedback we gather in this process. Since the original Hostmaster evolved into Aegir over a decade ago, many commercial Drupal hosting providers (Acquia, Pantheon, Platform.sh, etc) have come to the fore and serve a valuable role in the ongoing growth of Drupal on the web. As strong open source advocates, we believe it’s important to have an open, free alternative to commercial providers where we can address the additional use-cases in this space (eg. self-hosters, SaaS owners, etc.) and to help keep the Drupal devOps community open and growing. Watch this space for an upcoming series to talk in more detail about these use-cases and how we imagine the ongoing evolution of Aegir5 :) Consensus plans to continue being a community steward for Aegir, as we feel passionate about it’s value both to ourselves and as a community. We look forward to wider participation in this effort. Please get in touch if you’re interested in supporting Aegir5! Wish us luck at DrupalCon Pitch-burg! https://consensus.enterprises/blog/aegir5-kubernetes-backend/ Tue, 16 May 2023 09:00:01 -0500 Aegir5: Kubernetes Backend integration on Consensus Enterprises Blog published Tue, 16 May 2023 09:00:01 -0500 In previous posts we covered how the Frontend and queue mechanisms can talk with the Backend. We also covered the stand-alone work we’ve been doing within Drumkit to support Drupal on Kubernetes. In this post, we’ll discuss how we plan to integrate this new Backend into the existing Aegir 5 architecture. To integrate the Kubernetes Backend into Aegir 5, we will need to build new top-level entities (see this earlier post about Clusters, Projects, Releases, and Environments) for the Frontend. … In previous posts we covered how the Frontend and queue mechanisms can talk with the Backend. We also covered the stand-alone work we’ve been doing within Drumkit to support Drupal on Kubernetes. In this post, we’ll discuss how we plan to integrate this new Backend into the existing Aegir 5 architecture. To integrate the Kubernetes Backend into Aegir 5, we will need to build new top-level entities (see this earlier post about Clusters, Projects, Releases, and Environments) for the Frontend. These entities are composed almost entirely of Operations. The Tasks on the Frontend will primarily gather the variables needed to generate the templates in Drumkit on the Backend. These are passed through the Queue system to the Backend. The Backend in turn instantiates the configuration files, and runs the appropriate kubectl commands to deploy the various components into the Kubernetes cluster. Drumkit currently has make targets that initialize an Environment and deploy a Project Release to it, among other things. We will need to add a Drumkit Backend to extend dispatcherd to issue these make targets, passing in the Frontend variables. Having dispatched the job, the Backend will then gather output and send it back to the Frontend. In previous editions of Aegir, we sometimes saw the split-brain problem arise as a result of the Frontend and Backend sharing responsibility for the state of the system. In Aegir5, the Frontend is intended to be canonical with regards to the data about the state of a running application. Aegir manages the Environment, consisting of the application state (files + database). The Project incorporates the infrastructure Kubernetes resources required to host the application. Drumkit provides commands to initialize a project with such resource definitions. We are working on getting more specific about the tasks required to implement this in a new roadmap ticket. Our next post in the series will compare Aegir3 and Aegir5 from a features standpoint, after which we’ll discuss the plan to get from here to there, once we’ve established a clear roadmap. https://consensus.enterprises/blog/aegir5-queue-architecture/ Wed, 26 Apr 2023 09:00:00 -0500 Aegir5: Queue Architecture on Consensus Enterprises Blog published Wed, 26 Apr 2023 09:00:00 -0500 Previously in this series, we looked at the Aegir5 front-end interface architecture, as well as the lower level entities, Tasks and Operations that provide building blocks. As mentioned in the first part of the series, our most recent work on Aegir5 itself has been reworking the queue system. In this post, we explore this topic in more detail. The Aegir5 queue is implemented using Celery, which is a full-featured Python-based task queue, built atop RabbitMQ. Initially we built dispatcherd which … Previously in this series, we looked at the Aegir5 front-end interface architecture, as well as the lower level entities, Tasks and Operations that provide building blocks. As mentioned in the first part of the series, our most recent work on Aegir5 itself has been reworking the queue system. In this post, we explore this topic in more detail. The Aegir5 queue is implemented using Celery, which is a full-featured Python-based task queue, built atop RabbitMQ. Initially we built dispatcherd which operates as the back-end queue worker. For a variety of reasons that we’ll get into later, we also added relayd to run on the front end. dispatcherd is the Aegir5 equivalent of Drush Provision. It is responsible for writing configuration files and then running the commands that will instantiate whatever the configurations represent. dispatcherd runs commands such as ansible playbook. It then gathers the terminal output and sends it back to the front-end. Initially, this was done using an HTTP end-point, which would update the log record of the relevant operation. This worked reasonably well, but it required HTTP access from dispatcherd to the front end. This is where relayd comes into play. By hitting the front end with updates from the terminal output, we had to bootstrap the web UI repeatedly, putting unnecessary load on the front end. So we built an alternative solution: relayd running as a queue worker on the front end. This front end worker is very lightweight, picking up data and passing it on to an aegir:console command that can be more efficient at injecting the log results from the back end. Once we had added relayd, we realized it presented another opportunity. Up to this point, the front end had been posting directly into the queue for dispatcherd. However, this required the front end to have credentials to the Celery queue. Since the intention was for relayd to run in the same context as the front end, we could instead simply have the front end communicate with relayd securely without providing credentials to the back-end. In addition to a more loose coupling of all the components, this represents an improved security posture for Aegir5 overall. By removing queue credentials from the front-end UI, we reduce the risk of a vulnerability in the Drupal site leading to a compromise of the back-end infrasructure. Having built and tested the queue architecture fairly thoroughly we are poised to tie everything together. In the next post, we explore the integration of our Kubernetes framework with the queue mechanism and existing front-end UI to create a fully functional (albeit MVP) Aegir5 implementation. https://consensus.enterprises/blog/aegir5-frontend-ui/ Tue, 18 Apr 2023 09:00:00 -0500 Aegir5: Front-end UI architecture on Consensus Enterprises Blog published Tue, 18 Apr 2023 09:00:00 -0500 In our previous post, we looked at the Tasks and Operations which form the building blocks for the user interface in Aegir5. Here we’ll look at the additional entities required to support the Kubernetes-based backend framework. It is worth noting that Aegir has always had a tension between Developer and SysAdmin use-cases. We’ll cover this in more depth in a later post. For the moment, we’re focused on the Developer use-case. One thing to bear in mind is that Kubernetes operates quite … In our previous post, we looked at the Tasks and Operations which form the building blocks for the user interface in Aegir5. Here we’ll look at the additional entities required to support the Kubernetes-based backend framework. It is worth noting that Aegir has always had a tension between Developer and SysAdmin use-cases. We’ll cover this in more depth in a later post. For the moment, we’re focused on the Developer use-case. One thing to bear in mind is that Kubernetes operates quite differently than configuring and operating Nginx and MySQL directly, as we do with Aegir 3 Provision. As a result, the types of entities we are working with are going to be somewhat different. However, they will largely map to existing concepts in Aegir 3. As seen in the diagram from the Kubernetes backend post (reproduced below), the core entities are dealing with are Clusters, Projects, Releases and Environments that we’re managing. N.B. The terms below are how we’re currently describing the high-level components. Our goal here is to make these as natural and clear as possible. This may evolve over time, especially if we have to continuously explain what any of them mean (i.e. “Platform”). Clusters, in this case, map fairly well to Servers. We intend to support provisioning of Clusters in the future. For the time-being, we will simply be pointing at existing Clusters to which we will want to deploy our applications. Projects represent the codebase at a more abstract level. Whereas Releases (see below) are fully instantiated codebases, Projects represent the source of the codebase itself; usually a Git repository. We may introduce a similar entity type, Applications, down the road. These would serve a similar purpose, but optimized for “cattle” in the SysAdmin use-case. Devshop and the Aegir Distributions module provide similar functionality in Aegir 3. Releases are basically the equivalent of Platforms in Aegir 3. They capture the runtime environment, with a specific version of our codebase deployed and all dependencies downloaded. In Kubernetes, these are basically container images. Sharing a common codebase across multiple sites can be achieved by leveraging the same Release. However, unlike Platforms, we are not intending to support Drupal’s multisite capability in Aegir5. Multisite adds a lot of complexity, especially when running in a containerized environment. It also introduces multiple security and stability concerns. Environments become the equivalent of Sites in Aegir 3. They capture the specific configuration required to run an instance of a specific Release of the Project (or Application), along with the storage required for both the database and any uploaded files. These are what we believe to be the basic building blocks that we need to wire up to the Drumkit Kubernetes backend. We’ve tried not to bake in too tight of a coupling to the Kubernetes backend. They ought to be re-usable for the Ansible backend, as well. Our medium-term goal is to build a robust, reliable and flexible continuous delivery system. Some of this will likely require much more opinionated approaches. We want to keep the basic components minimally opinionated, so that they can be composed into more elaborate workflows to satisfy different use-cases (eg. SaaS operations). So far in this series we’ve looked at the Kubernetes framework we’ve built which will form the new backend for Aegir5. We’ve also explored the Drupal content architecture that forms the front-end. In the next post we’ll look at the Queue architecture which ties the front-end and back-end together before moving on to our Roadmap for bringing the whole thing together. https://consensus.enterprises/blog/aegir5-frontend-low-level/ Thu, 06 Apr 2023 09:00:00 -0500 Aegir5: Front-end low-level architecture on Consensus Enterprises Blog published Thu, 06 Apr 2023 09:00:00 -0500 In our previous post, we talked about our recent client work building a Kubernetes-based system for hosting web applications. We’ve defined a general framework to support our development and production hosting workflows, and recognized this as a solid basis for an alternate backend to plug in to the existing Aegir5 front-end. Today we’ll take a look at the Drupal architecture underlying that front-end. In Aegir5, the building blocks consist of Task and Operation entities. Tasks represent … In our previous post, we talked about our recent client work building a Kubernetes-based system for hosting web applications. We’ve defined a general framework to support our development and production hosting workflows, and recognized this as a solid basis for an alternate backend to plug in to the existing Aegir5 front-end. Today we’ll take a look at the Drupal architecture underlying that front-end. In Aegir5, the building blocks consist of Task and Operation entities. Tasks represent relatively low-level reusable functionality. For example, we might have a Task for the Ansible back-end that is responsible for writing an Nginx vhost. We might then bundle that Task, along with others, in order to compose an “Install Drupal” Operation. Since the need to write a vhost is pretty common for web applications, that same Task might also be bundled into an “Install Joomla” Operation. Operations represent the actions that a user might want to trigger. Operations are configured to contain one or more Tasks. Each Task is a standalone fieldable entity. As a result each Task is responsible for gathering whatever data it needs to run its back-end tasks. The Operation includes these Tasks as inline entities. When we trigger an Operation, we are presented with a form that includes all the fields from the various Tasks associated with that Operation. These values, across all of the related Tasks, are marshalled and sent to the back-end. Logs are kept at the Operation level and so the output from the back-end gets piped back into a field on the Operation. In addition to the terminal output from the back-end, we also include the return code of the operation, so that we can record whether it succeeded or not. Another type of task is what we call an “action” task. These do not get bundled into Operations, but rather happen within a given page request. For example, instead of explicitly triggering a “password reset” task, as we do in Aegir 3, we can have a task dispatched when we click a “Log into site” button. This task runs a drush uli (via the Celery task queue), and relays the result directly back to the front-end, almost immediately. This can in turn redirect us directly to that URL. The result is that we are logged into the site without having to wait for a fully logged Operation to run. In the same way that Operations are made up of Tasks via inline entities, other entities such as Sites and Platforms are themselves made up of Operations embedded as inline entities. When we create a Platform entity, we instantiate all of its Operations and related Tasks. Among other things, this allows us to reuse Tasks between Operations. For example, when, taking a backup, we will want to record the file path to the tarball. This data then becomes available to the restore task that may come later. This case also illustrates the need for the back-end to be able to write data to the front-end. This is another function of relayd in the queue architecture. (see the upcoming post on this for more details). In addition to writing logs to Operation entities, it can write data to any field within any entity on the front-end. The sequence diagram below illustrates the flow from an Operation being triggered to run, through placing a task on the queue, running the backend executor (currently Ansible, soon Kubernetes), and getting feedback returned to the frontend. This architecture largely mirrors how Aegir 3 works. However, it provides a more flexible, robust and scalable solution. In our next post, we’ll take a look at the higher-level UI architecture we plan to implement to represent the key components in our Kubernetes hosting framework: Clusters, Projects, Releases, and Environments. https://consensus.enterprises/blog/aegir5-kubernetes/ Mon, 20 Mar 2023 09:00:01 -0500 Kubernetes backend for Aegir5 on Consensus Enterprises Blog published Mon, 20 Mar 2023 09:00:01 -0500 Lately we’ve been working with clients ranging from large Canadian government departments to small commercial SaaS companies, who have asked us to deploy CMS apps to Kubernetes (K8S) clusters running on Openstack. In spite of our continued feeling that most of the time Kubernetes Won’t Save You, we’ve found it to be surprisingly useful in certain contexts. In fact, we’ve started to think that K8S will prove an extremely valuable backend to plug in to our existing Aegir5 front-end and queue … Lately we’ve been working with clients ranging from large Canadian government departments to small commercial SaaS companies, who have asked us to deploy CMS apps to Kubernetes (K8S) clusters running on Openstack. In spite of our continued feeling that most of the time Kubernetes Won’t Save You, we’ve found it to be surprisingly useful in certain contexts. In fact, we’ve started to think that K8S will prove an extremely valuable backend to plug in to our existing Aegir5 front-end and queue system. We started developing this system for a project that required a Django app paired with a React frontend. This presented an opportunity to prototype a system capable of hosting a fairly complex application. Since then, we have deployed multiple Drupal sites in a similar fashion. This transition was surprisingly straight-forward; partly due to not needing to support headless scenarios, and thus being able to simplify quite a bit. As shown in the diagram below, traffic enters the system via the router that directs traffic to the cluster. The router exposes a public IP address, to which we point our DNS entries. From there the K8S ingress service directs traffic within the cluster. In addition, the ingress controller automatically generates Let’s Encrypt HTTPS certificates, and acts as the HTTPS endpoint, handling the TLS handshake, etc. Traffic gets directed to the Drupal deployment, which in turn, connects to the database deployment. These are running Docker containers. In addition, temporary job containers can be launched to perform specific tasks, such as installing the site, or running cron. In the case of the Drupal containers, we’re running a custom Drupal image that bakes our code base in. The database deployment, for its part, is using a stock mariadb image. Our custom Drupal image, by including the project code base, provides us with reproducible builds. We can deploy this same image to multiple environments with confidence that it will be consistent across all of them. Both the database, and the Drupal site are connected to storage via Persistent Volume Claims (PVCs). More on this later. We use Terraform to deploy the clusters themselves to OpenStack service providers. With Terraform, we can define the size of the Kubernetes clusters that we want to run and launch them programmatically. Because the initial application was intended to be a SaaS product, we ended up engineering a multi-cluster multi-environment architecture. For this project, we had a development cluster and a production cluster. We have since renamed them “unstable” and “stable” respectively, to indicate which types of environments are appropriate to host on each. Our client was targeting their products at large financial institutions. As such, we were anticipating fairly stringent security requirements. Each financial institution wants their client data isolated from any other users of the system. By supporting multiple clusters and multiple environments per cluster, we provided a multi-tenant system that could isolate data at a number of different levels. While that capability is certainly interesting, we have primarily been implementing and using simpler development workflows. In our Drupal projects, we have primarily used a single cluster with testing, staging and production environments. These environments all run within a single Kubernetes cluster. However, they isolate resources effectively by using K8S namespaces. Once we have a cluster running, most of the workflows happen within K8S. We keep these logically segmented using namespaces. Together with dedicated storage, namespaces define environments. Within these environments, we deploy the configuration for the Drupal app itself. In designing this architecture, we also wanted to take into account persistent data. In both the Django and the Drupal scenarios, we had data being stored in a database and files that were uploaded into the application and stored on the filesystem. In order for this to persist between rebuilds of the application itself, we extended our concept of an environment to include not only the name space, but also the Persistent Volume Claims (PVCs), where we were storing this data. These are all defined as Kubernetes resources, but are handled separately, so as not to never mistakenly delete files or data. This architecture basically comes down to generating various configuration files and instantiating the resources that they describe. We have wrapped these in Drumkit (GNU Make) targets that template the configuration files (using Mustache), and run all necessary commands in the correct order, with the appropriate options and arguments. This mirrors the functionality of Provision in Aegir. Our intention is to use this system as a backend for Aegir5. Watch for an upcoming blog post which tackles our planning and vision for next steps! https://consensus.enterprises/blog/aegir5-is-happening/ Sat, 25 Feb 2023 09:00:00 -0500 Aegir5 Development is Happening! on Consensus Enterprises Blog published Sat, 25 Feb 2023 09:00:00 -0500 Aegir5 development is happening! We (Consensus) have been making steady progress on it over the last few years and are looking to kick off a new burst of focused development. Here’s a summary of progress that has been made so far and how you can contribute. First off, as you’re probably aware, Aegir5 is a complete re-write of Aegir. We are intending to build on all the great aspects of Aegir, while freeing ourselves from a codebase that is rooted in PHP 4. We’re using D9+ alongside modern … Aegir5 development is happening! We (Consensus) have been making steady progress on it over the last few years and are looking to kick off a new burst of focused development. Here’s a summary of progress that has been made so far and how you can contribute. First off, as you’re probably aware, Aegir5 is a complete re-write of Aegir. We are intending to build on all the great aspects of Aegir, while freeing ourselves from a codebase that is rooted in PHP 4. We’re using D9+ alongside modern middleware and back-end components. The front-end (Drupal) and the queue system (based on Celery) are working fairly well. We’ve mostly reproduced the Aegir 3 behaviour, and it has a similar look and feel. This is the most recent work that’s been done directly in the Aegir5 repo, and dates back about a year, or more. (See for example aegir#94.) It’s suffering from some bitrot, so we’ll need to do a bit of cleanup. Bear in mind that we’re self-funding this project, so we aren’t currently able to dedicate much time and budget to it directly. However, we have had a number of contracts, both current and in the recent past, where we’ve been able to evolve some of the middleware and back-end components. For example, we recently worked with the Drupal Association to build Rugged, a TUF server implementation to secure the Drupal Composer repository. The Drupal Infrastructure team is in the process of beta testing this functionality on drupal.org, and is very much in line with the Aegir project’s goals of making updates easy and secure. More directly related to Aegir5, Rugged uses Celery extensively, and started out building from the components we’re using in Aegir5. We’ve developed them much further within the context of that project, and plan to port those improvements back to Aegir5. Likewise, our initial plan was to build the Aegir5 back-end using Ansible. While we have some basic playbook elements in Aegir5 currently, we’ve had a number of clients recently ask us to build fully automated Kubernetes-based hosting systems. Our first prototype is hosting a Django/React app, and is working well in production. We’ve been hosting our Kubernetes clusters on OpenStack, since we’ve been striving for a fully FLOSS stack as part of our overall goals for Aegir5. However, a number of our larger institutional clients are restricted to using one of the major cloud vendors. So we’ve been architecting our automation to support any Kubernetes cluster. We’re currently refining these components for a Drupal upgrade project for a department of the Government of Canada. The Drupal-on-Kubernetes piece was mostly completed a couple weeks ago. We’re continuing to enhance and stabilize it. Next, we’re planning to integrate something along those lines as the default back-end for Aegir5. We do not have a timeline for the integration of these components into Aegir5, mostly due to the aforementioned time/budget constraints. However, we are eager to proceed with that work. We have had a handful of organizations express interest in possibly funding some of this upcoming work, some of whom are currently hosting several hundred Drupal sites each on Aegir 3. If you know of someone who might be interested in helping to fund this work, please let us know. We’re undertaking to update the Aegir5 roadmap to reflect the progress we’ve made, and to outline next steps. Ideally we’ll be able to get a detailed enough break-down to be able to broadly estimate remaining effort for an alpha release. Hopefully this gives a good picture of the current state of Aegir5 development. We’ll be publishing some subsequent posts in the coming days to describe in more detail our Drupal-on-Kubernetes architecture, as well as how we aim to integrate the existing frontend and queue system work to achieve an MVP Aegir5. If you have any questions, don’t hesitate to ask. Aside from our Contact form, you can find us on LinkedIn or Matrix, and always eager to talk about Aegir! https://consensus.enterprises/blog/moving-terraform-state-from-openstack-swift-to-gitlab/ Mon, 09 Jan 2023 09:00:00 -0400 Moving Terraform State from OpenStack Swift to GitLab on Consensus Enterprises Blog published Mon, 09 Jan 2023 09:00:00 -0400 For our cloud computing, we typically use an OpenStack provider because of its open-source nature: There’s no vendor lock-in, and the IaaS code is peer-reviewed unlike providers such as AWS, Azure, GCP, etc. (Shout out to Vexxhost for having great support!) As such, we’ve been using OpenStack’s Swift object storage service for storing Terraform’s state, which allows Terraform to track all of the resources it manages for automating infrastructure. Recently, however, support for the Swift backend … For our cloud computing, we typically use an OpenStack provider because of its open-source nature: There’s no vendor lock-in, and the IaaS code is peer-reviewed unlike providers such as AWS, Azure, GCP, etc. (Shout out to Vexxhost for having great support!) As such, we’ve been using OpenStack’s Swift object storage service for storing Terraform’s state, which allows Terraform to track all of the resources it manages for automating infrastructure. Recently, however, support for the Swift backend has been removed. If you’re still using Swift for this purpose, you’ll need to migrate your Terraform state files to another backend. Because the official migration documentation is sparse, I’ll describe how to migrate from Swift to GitLab-managed Terraform state. GitLab is a fantastic option because it can be used to manage so many other aspects of your project that you need anyway: Git repository hosting, issue tracking, CI/CD, etc. We use GitLab for all of our projects so it’s a great fit for us. The actual step of migrating the data is well supported, but there’s some required set-up before and after. When you change backends, Terraform gives you the option to migrate your state to the new backend. This lets you adopt backends without losing any existing state. Downgrade to the latest pre-1.3 Terraform version. e.g. sudo apt install terraform=1.2.9 Navigate to your Terraform directory. cd /path/to/git/repository/terraform Move your local state files out of the way as they could be set up for a different environment. mv .terraform /tmp Set up your environment variables to connect to use your existing state backend. e.g. source ../openstackrc/vexxhost-...-staging-ca-ymq-1.openrc.sh Initialize Terraform from the remote state. terraform init Back up your current state. cp .terraform/terraform.tfstate terraform.tfstate.backup-staging In your Terraform code, in your backend stanza, replace swift with http. Unset the old state environment variables. export TF_CLI_ARGS_init= Fetch one of your Gitlab personal access tokens with the api permission. If you don’t have any that aren’t expired, create a new one in your settings. To actually migrate the data, the GitLab documentation says to set a single environment variable, and then manually run terraform init with many options. Given that this is error-prone and not easily repeatable, I’d recommend using a shell script (or similar) instead. Create a file named setup-terraform-variables, and populate it like so: #!/usr/bin/env bash ############################################################################# ## Set up Terraform variables. ############################################################################# # Set up the Gitlab.com state backend. export OS_PROJECT_CLOUD="$OS_PROJECT_DESCRIPTION-$OS_PROJECT_ENVIRONMENT-$OS_REGION_NAME" export TF_GITLAB_PROJECT_ID="" echo "Please enter your gitlab.com username: " read -r TF_GITLAB_USERNAME export TF_GITLAB_USERNAME echo "Please enter your gitlab.com personal access token: " read -sr TF_GITLAB_PASSWORD export TF_GITLAB_PASSWORD export TF_STATE_ADDRESS="https://gitlab.com/api/v4/projects/${TF_GITLAB_PROJECT_ID}/terraform/state/${OS_PROJECT_CLOUD}" export TF_CLI_ARGS_init="\ -backend-config='address=$TF_STATE_ADDRESS' \ -backend-config='lock_address=$TF_STATE_ADDRESS/lock' \ -backend-config='unlock_address=$TF_STATE_ADDRESS/lock' \ -backend-config='username=$TF_GITLAB_USERNAME' \ -backend-config='password=$TF_GITLAB_PASSWORD' \ -backend-config='lock_method=POST' \ -backend-config='unlock_method=DELETE' \ -backend-config='retry_wait_min=5'" You can set other Terraform variables in here as well, and include it in other deployment-environment-specific shell scripts that you run to set up each one. For example, if you’re using OpenStack generally, these would be your openstackrc files, which contain your credentials for accessing the API. For a further optimization, you can write the GitLab credentials to a local file so as not to have to enter them every time, but I’ll leave this as an exercise to the reader. (If I get a chance, I’ll come back here and update it.) In your Terraform configuration files, it’s necessary to change the backend type from Swift to HTTP. terraform { - backend "swift" { + backend "http" { # Must be read from environment variable `TF_CLI_ARGS_init` because normal # variables cannot be used here. } } If you’re wondering why we need to use TF_CLI_ARGS_init, and can’t use Terraform variables in the stanza, see my earlier article Setting Deployment Environments’ Terraform State Backends with Environment Variables . You can now run: terraform init You should now see something like this, which requires your confirmation part-way through. Initializing the backend... Terraform detected that the backend type changed from "swift" to "http". Acquiring state lock. This may take a few moments... Acquiring state lock. This may take a few moments... Do you want to copy existing state to the new backend? Pre-existing state was found while migrating the previous "swift" backend to the newly configured "http" backend. No existing state was found in the newly configured "http" backend. Do you want to copy this state to the new "http" backend? Enter "yes" to copy and "no" to start with an empty state. Enter a value: yes Releasing state lock. This may take a few moments... Successfully configured the backend "http"! Terraform will automatically use this backend unless the backend configuration changes. Initializing provider plugins... [...] Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. This will purge your old state, and set it up to match the new remote state. Move your local state files out of the way as they reflect the old state backend. mv .terraform /tmp Upgrade Terraform to the latest version. e.g. sudo apt install terraform Set up your environment variables again, using the updated code, to connect to your desired cloud environment. This script must include setup-terraform-variables as discussed above. source ../openstackrc/vexxhost-...-staging-ca-ymq-1.openrc.sh Initialize Terraform from the remote state. terraform init You can now see any of your state files in the GitLab Web UI on your Gitlab project’s page. Simply navigate to Infrastructure - Terraform, and they’ll be listed. https://consensus.enterprises/blog/setting-environments-terraform-state-backends-with-environment-variables/ Mon, 12 Dec 2022 14:00:00 -0400 Setting Deployment Environments' Terraform State Backends with Environment Variables on Consensus Enterprises Blog published Mon, 12 Dec 2022 14:00:00 -0400 Terraform is an essential tool for automating cloud-computing infrastructure and storing it in code (IaC). While there are several ways to navigate between deployment environments (e.g. Dev, Staging & Prod), I’d like to talk about how this can be done with environment variables, and explain why it can’t be done more naturally with Terraform variables. I originally wrote this as a comment on the feature request Using variables in terraform backend config block, which explains Terraform’s design … Terraform is an essential tool for automating cloud-computing infrastructure and storing it in code (IaC). While there are several ways to navigate between deployment environments (e.g. Dev, Staging & Prod), I’d like to talk about how this can be done with environment variables, and explain why it can’t be done more naturally with Terraform variables. I originally wrote this as a comment on the feature request Using variables in terraform backend config block, which explains Terraform’s design limitations preventing it from allowing any variables in backend configurations, which is what Terraform uses to determine where to store its state data files (which track the cloud resources it manages). To illustrate, this is where variables can’t be used, even though the configuration must change when the environment changes: terraform { backend "whatever" { [...] } } Alternatively, it’s possible to have a single backend state configuration that stores data for multiple environments using Workspaces. You can then use Terraform CLI commands to switch between them. However, this solution is not not appropriate for all use cases, as per the documentation: Workspaces are not appropriate for system decomposition or deployments requiring separate credentials and access controls. If this isn’t a problem for you, Workspaces is a good option. Otherwise, please keep reading. In demonstrating a solution, I’m going to be working with OpenStack, which is the IaaS run by many cloud providers, but the same concept can be applied to other IaaS, such as AWS, Azure, GCP, etc. Here, the backend state data will be stored in OpenStack’s Swift object storage service. Update (2023-04-11): While the example below is still valid conceptually, Swift can no longer be used as a Terraform backend because support for it has been removed. If you’re still using Swift for this purpose, the official migration documentation is sparse. However, I provide guidance in a more recent article: Moving Terraform State from OpenStack Swift to GitLab. Typically, you’d download an OpenStack RC file, which contains the credentials needed to access the API. In each of these (e.g. vexxhost-abc-prod-ca.openrc.sh, where the format is provider-project-environment-region), I append the following lines to the end: [...] ############################################################################# ## Additional set-up not included with IaaS provider's OpenStack RC files ### ############################################################################# export OS_PROJECT_ENVIRONMENT="abc-prod-ca" source $(dirname "$0")/setup-terraform-variables.sh The sourced (included) file then looks like: #!/usr/bin/env bash ############################################################################# ## Set up Terraform variables. ############################################################################# # Set up backend state container names. export OS_PROJECT_STATE="$OS_PROJECT_ENVIRONMENT-terraform-state" export OS_PROJECT_STATE_ARCHIVE="$OS_PROJECT_STATE-archive" export TF_CLI_ARGS_init="\ -backend-config='container=$OS_PROJECT_STATE' \ -backend-config='archive_container=$OS_PROJECT_STATE_ARCHIVE'" # You can set other TF variables in here as well. echo "Please enter the outgoing e-mail account's password: " read -sr TF_VAR_smtp_password_unquoted export TF_VAR_smtp_password="\"$TF_VAR_smtp_password_unquoted\"" The special environment variable here is TF_CLI_ARGS_init. This is what Terraform uses to configure the backend, if it’s set. So by changing that environment variable, you can change the backend configuration. All that’s needed in the code is the following, basically just the backend type, with details being pulled in from the environment. In main.tf (or wherever), the backend definition: terraform { backend "swift" {# Must be read from environment variable `TF_CLI_ARGS_init` because normal # variables cannot be used here. } } To switch between environments, I simply source another OpenStack RC file; I have one for every environment that I care about. With the above set-up, it also switches the Swift object storage containers used for backend state. You can then run terraform init. I’ve found that sometimes it’s necessary to delete your .terraform directory before rerunning terraform init though, or it’ll get confused and you’ll get strange error messages. So as standard practise, I run the following command to remove it first: mv .terraform /tmp You cannot use Terraform variables to vary the backend state configuration. You can, however, use the environment variable TF_CLI_ARGS_init instead. Run a script to set this environment variable with the configuration matching the deployment environment you’d like to work with. Simply specify the backend type in the Terraform code, and TF_CLI_ARGS_init along with your script(s) will take care of the rest. https://consensus.enterprises/blog/october-2022-retreat/ Mon, 31 Oct 2022 09:00:00 -0400 Consensus Company Retreat - October 2022 on Consensus Enterprises Blog published Mon, 31 Oct 2022 09:00:00 -0400 Last week our fully remote team gathered in person at a beautiful spot in Prince Edward County, Ontario, for our second annual company retreat. Our first iteration in 2021 was a resounding success, and we decided to do it again this year. In both cases, we took advantage of the high-bandwidth nature of being together in physical space to hold a variety of big picture conversations about our accomplishments, direction and goals. As we try to do in all our work, we sought to iterate and improve … Last week our fully remote team gathered in person at a beautiful spot in Prince Edward County, Ontario, for our second annual company retreat. Our first iteration in 2021 was a resounding success, and we decided to do it again this year. In both cases, we took advantage of the high-bandwidth nature of being together in physical space to hold a variety of big picture conversations about our accomplishments, direction and goals. As we try to do in all our work, we sought to iterate and improve on last year’s retreat. This post will review some of the key differences we saw at the retreat, as well as highlight some of the organizational changes we noted in the past year. The first difference that was very noticeable this year is the increased participation across the team. The 2021 retreat was largely facilitated by the organizers who had crafted the agenda. This year we had a completely different group of organizers, and each department circle was consulted for session ideas. We then asked circles to articulate goals for the session, and to appoint a facilitator. This led to more voices at the front of the room, as well as more interactive participation across the team. We grouped the sessions around the themes of Past, Present, and Future, which resulted in a clear progression from one discussion to the next. On top of all that, our team has gained a couple new members since last year, so we were able to hear from brand new voices with fresh perspectives as well. We even made space for remote participation in sessions for the few people who were unable to attend everything in person. Probably the biggest organizational change we made in the last year was to implement and mature our sociocratic governance model and organizational structure. During the 2021 retreat we had just begun to understand sociocracy as a governance model. We spent a lot of time and effort over the last year learning, trying, implementing, and maturing our circle structure, aims and domains, and processes. Coming out of the 2021 retreat, we had brainstormed a list of core values and used these to craft Vision and Mission statements. In the ensuing months, we refined and reworked the core values into a coherent list, and then set about defining Aims of the organization that aligned with those things. In tandem, we evolved our circle structure as we noticed gaps or inconsistencies. We added circles and adjusted the relationships between circles to improve the effectiveness and efficiency of running the company. We also developed a better understanding of the key meeting roles (facilitator, secretary, leader, delegate) in the sociocracy framework, as well as operational roles in specific circles like Project Lead in our project Delivery circle. At this year’s retreat, it became obvious that we had developed some healthy sociocratic habits that organically translated into the sessions. We naturally assigned facilitator, secretary, and timekeeper roles, we spoke in “rounds” for most of our discussions, and sought consent wherever a key decision was needed. At the first retreat we were very focused on being in-person, and aimed to minimize screens and tech within the sessions. This proved very effective in terms of engaging discussions in the room, but later we realized that the flipchart notes we’d taken were woefully inadequate in terms of capturing the key points of discussion. This year we hit on a couple of improvements that seemed to strike a nice balance. Each session had a dedicated etherpad, pre-populated with the outline of the session and enough metadata to make it easy to publish as meeting minutes later. At the start of each session we asked for a volunteer to secretary, so that only one person needed to have a laptop for taking notes in the relevant pad. Meanwhile, we’d produced a well-designed, good quality bound paper workbook containing key items like our values, mission and vision, a description of each session, plus a blank page for taking notes in between. This allowed us to have a physical place to keep track of the agenda and schedule, and take written notes as desired. The workbook had a link to get back to the pads, and we simply asked attendees to transfer any important notes into the pad later to be captured in the minutes. Going into planning the retreat this year, we were keenly aware that last year’s retreat felt overly full with structured working sessions. We had ambitious plans to get alignment on an array of topics, and felt we needed to fill as much of the available time as possible to get through everything we had in mind. This year, we consciously aimed to leave much more space in the schedule to allow for social time and unstructured activity together. To accomplish this, we did a few things: Built out the time schedule first, with liberal breaks included, and then fit the sessions into the available time, rather than the other way around. Asked those proposing sessions to identify one or two clear goals for the session, to help focus discussion and identify when we were done. Assigned a dedicated timekeeper to keep us on track. In some cases we used very short 1-2m timers during rounds to help remind us of the passing of time. The result was that things felt a lot more spacious overall, and we had a lot more fun, social time together. In our wrap-up session where we started gathering feedback, this came up consistently as a positive aspect of the retreat. I came away feeling very good about doubling-down on this aspect. This was further validated when someone posted a link to this great blog post from Doist espousing the 20/30/50 rule for remote team retreats: 20% work, 30% activities, 50% free time. I think this will be a useful guideline for future iterations and improvements on Consensus retreats. https://consensus.enterprises/blog/gitlab_prioritizing_todo_list/ Tue, 27 Sep 2022 10:54:00 -0400 Managing priorities in Gitlab with the to-do list feature on Consensus Enterprises Blog published Tue, 27 Sep 2022 10:54:00 -0400 If you’ve ever done any software development, or even if you haven’t, you’ve probably heard of GitLab. It’s great for developing, securing and maintaining software, being a fairly complete DevOps toolbox. We actually use it for tracking all of our work items, even outside of software (with the exception of CRM, but they’re working on that). For example, we have projects in there for operations, business development, HR, etc. The issue boards are fantastic. One of the nice features it has is a … If you’ve ever done any software development, or even if you haven’t, you’ve probably heard of GitLab. It’s great for developing, securing and maintaining software, being a fairly complete DevOps toolbox. We actually use it for tracking all of our work items, even outside of software (with the exception of CRM, but they’re working on that). For example, we have projects in there for operations, business development, HR, etc. The issue boards are fantastic. One of the nice features it has is a to-do list, which gets populated automatically from others assigning things to you, or mentioning you in various places. You can also add items manually. I use it rather extensively to track the work I need to do (and non-work, as I also track my personal tasks there as well). If something’s not in my e-mail inbox, which I try to keep empty, I probably have a ticket for it in GitLab. Or rather, an “issue” as they call it. I’ve come up with a system for sorting my to-do list in priority order as I sometimes have trouble deciding what to do next. As a Partner in the firm, who’s often swamped with a variety of too many things to do, this can be tricky to navigate. So I thought I’d share my solution. For each project where you want to have items show up above unprioritized items, add the following scoped labels to the top-level group in Group information - Labels. (If you’re not on a paid plan, you may have to use regular labels instead, which aren’t as nice.) Priority::1 - Highest Priority::2 - Higher Priority::3 - High In the project itself, go to Group information - Labels, star them, and then ensure that they’re sorted properly. (It’s necessary until Add priority to group based labels is fixed.) For each to-do item, tag it with one of those three labels. (Due dates are helpful too as they’re displayed in the to-do list, but are unnecessary for this to work.) Navigate to your To-Do List, and change the sorting (on the right side of the filter bar) to Label priority. You should now see your prioritized items in order, listed above unprioritized items. What’s great about this approach is that it works across groups and projects so you can get a single to-do list for everything, e.g. client projects, internal work, personal tasks, etc. I hope this helps you as much as it’s helped me! https://consensus.enterprises/blog/tuf_for_humans/ Tue, 10 May 2022 09:00:00 -0400 TUF for Humans: Explaining software update security on Consensus Enterprises Blog published Tue, 10 May 2022 09:00:00 -0400 For the past few months, we’ve been working with the Drupal Association on a project to enhance the security of the Drupal.org software repository. The most succinct way of describing this project is: Securing automated software deployments from supply chain attacks. Recently, on a long drive with my mother, I tried to explain this project to her. She is probably the least technical person I know. This may sound like the common tech trope, but it’s not. This has nothing to do with her age or … For the past few months, we’ve been working with the Drupal Association on a project to enhance the security of the Drupal.org software repository. The most succinct way of describing this project is: Securing automated software deployments from supply chain attacks. Recently, on a long drive with my mother, I tried to explain this project to her. She is probably the least technical person I know. This may sound like the common tech trope, but it’s not. This has nothing to do with her age or gender. She literally self-identifies as a neo-luddite. Over the course of that conversation, and subsequent ones, we came up with some helpful explanations and analogies that provide some insight into a very complex subject. So I figured that I’d share them here. Below, we’re going to cover the following topics: Software Supply Chains Validating Signatures Validating Documents Software Package Updates Verifying Packages Public-key Cryptography Digital Signatures Signing Packages First off, a new type of software security threat has been on the rise: supply chain attacks. In fact, by some estimates, they tripled in the last year alone. Some have been fairly high-profile, such as the recent Log4j and SolarWinds attacks. But what is a “supply chain attack”? Here the “supply chain” basically refers to all the software components that are used to build a modern application. Basically, instead of attacking a specific organization’s IT infrastructure, this kind of attack tries to inject malicious code into one of those software components. If it’s successful, then any organization that uses the compromized component is potentially vulnerable. One approach to address this type of threat systematically is The Update Framework (TUF). In the TUF Specification, the authors outline a host of attacks and weaknesses that they’re trying to address. They then go on to outline a rigourous process to secure software updates against these threats. The Update Framework is quite complicated. Even people with lots of experience with security, cryptography and package management often find it difficult to understand. Below, we try to explain some of the basic problems the TUF Spec is trying to solve, and how it suggests that we can address them. To help illustrate the nature of the problem space, let’s first consider some real-world examples: chequing accounts and contracts. When you go to a bank, to open a chequing account, they’ll generally ask you to sign a signature card that they’ll keep on file. When you then write a cheque, the bank will compare your signature on the cheque to the one they have on file for you. They should only transfer money out of your account if they match. The bank is verifying the authenticity of your signature. Another common place we find signatures is on contracts. Contracts are most often between two parties, and so have two signatures. On occasion, one of the parties might try to get out of their contractual obligations by claiming that they never signed the contract. If we were to point to their signature on our copy of the contract, they may claim it’s a forgery. The technical term for this is “repudiation”. How do we overcome this problem? One common approach is to have witnesses also sign the contract. Witnesses aren’t bound by the terms of the contract itself. Instead, their signature is meant to prove that the other parties did, in fact, sign the contract. If a contract dispute were to go to court, the witnesses may be called to testify to that effect. Each witness is asserting the authenticity of the signatures. It is also common to revise a contract’s terms. This often happens repeatedly before finally signing the contract. But it can also happen after a contract was already signed. So now we have multiple versions of a contract. What happens if the parties can’t agree on which version is authoritative? To avoid this problem, for really important contracts, a professional notary may be hired. The notary would review all copies of the most recent version of the contract and apply their seal and signature to each of them. The notary also keeps a logbook, where they register each time they notarize a document. These provide proof that all of the copies of the contract are identical. A notary’s seal is embossed into the paper of each copy of the contract. Only the notary has access to their seal. Older or altered versions of a contract would not have this seal. So they wouldn’t be considered valid. So the notary’s seal is validating the authenticity and integrity of the document (contract). These are simplified descriptions of these processes. But they should illustrate how signatures and seals can help to authenticate and verify a document. Let’s see how these principles can be applied to improving software security. Software security is, itself, a complicated subject. So we’ll start by describing a simplified example. Imagine that I have a simple blog website. Despite its outward simplicity, this website runs atop many, many components. We refer to this as a “software stack”. It’s a simplified version of the “software supply chain” that we looked at earlier. Such a blog website will likely be: Hosted on a server, probably running a Linux operating system; Served to your browser by a web server, such as Apache or Nginx; Written in a language like PHP or Python; Storing content in a database, such as MariaDB or Postgres; Built on a web framework like Drupal or Django; and Using various extensions and modules, to provide specific functionality. There are many undiscovered bugs and security flaws in the thousands of components that make up this stack. To keep my website secure, I need to promptly update any component that releases a new version that fixes a security flaw. Otherwise, if I leave the insecure code unpatched, an attacker might exploit it. For the sake of argument, let’s assume that my website was built using Drupal. Luckily, Drupal has a spectacular security team. They publish regular security advisories on a mailing list to which I’m subscribed. So I get alerted whenever I need to update my codebase. When I receive an alert that a module I’m using has a new security release, I’ll use a tool like Composer to download the latest version of the module and deploy the update to my website. After a couple more steps, my website has been updated, and the bug fixed. So far, so good… Except, attackers are getting increasingly sophisticated in how they go about compromising website security. What if, when I downloaded the new version of the module, someone managed to intercept the request, and instead delivered a hacked version of the module. This is a “Person-in-the-Middle” (PITM) attack. Now I’m actually worse-off than I was before, and I wouldn’t even know it. This is an example of what we defined earlier as a “software supply chain attack”. There are a number of similar types of attacks. For example, a “replay” attack is one in which, instead of downloading a hacked version of a module, I instead download an older version. This can be just as detrimental, and might be even harder to detect. But for the sake of simplicity, let’s just use PITM for our example here. What we need is a way to validate the authenticity, integrity and freshness of the packages that we’re downloading. This is where TUF comes in. Two components are required to verify package integrity: A TUF server, to sign packages (more on this below), and A TUF-enabled client, to verify the package signatures. Luckily, there’s a Composer plugin available to help with the second part. In our example, during the PITM attack, Composer can (via the TUF plugin) tell that the module has been tampered with. It can then display an error message, so that we’re aware of the problem, and react accordingly. The Composer TUF plugin basically steps in every time Composer downloads a file, and verifies the file’s integrity, freshness, etc. It does this by downloading information about the file, referred to as “TUF metadata”. This metadata includes information about the file, such as how big it should be. Most importantly, it contains a “hash” of the file. A hash is a long string of letters and numbers that uniquely identify a specific file. If you change even a single character in the file, then you’ll get a completely different hash. This hash acts like the notary’s seal. It allows us to easily verify that a given file has not been tampered with. But attackers are smart. How do we know they haven’t also altered the TUF metadata, to cover their tracks? This is where public-key cryptography and digital signatures come in. Here things start to get really tricky. Public-key cryptography relies on some very advanced math, and the way it works is quite unintuitive. As such, we’re only going to try to explain the most important element: asymmetry. Let’s start by looking at another real-world example: mail boxes. If you want to send me a letter, all you need is my address. However, for anyone to read that letter (once delivered), they’d need the key to my mailbox. Unlike my address, which is publicly available, I’m the only one with a key to my mailbox. This situation is asymmetric, meaning that it cannot be done in reverse. My mailbox key doesn’t help me to send a letter to you. Public-key cryptography works in a similar manner. It all starts by generating a “key pair”. A key pair consists of two files each containing a long string of characters (a cryptographic key). However, not just any two keys form a key pair. They need to be generated together, and have a very special relationship. If we encrypt a message with one of the keys, then it can only be decrypted by the other key in the pair. Notably, the key that was used to encrypt the message won’t help us to decrypt it. This holds true for both keys. Regardless of which one we use to encrypt a message, only the other one can be used to decrypt it. When we generate a keypair, we designate one of the keys as “public”, meaning it can be shared freely with others. We designate the other key “private”, meaning that it should never be shared at all. The public key acts like my address, in the example above. It allows anyone to encrypt a message that only I can decrypt, because I’m the only one with access to the private key. Digital signatures are basically the application of this same principle, but in reverse. Recall that either key can be used to encrypt a message that only the other key can decrypt. So, if I encrypt a message using my private key, then only the public key will be able to decrypt it. Remember that I’m the only person with access to my private key. So if I send you an encrypted message, and you can decrypt it with my public key, then you can be pretty sure that I’m the one that sent the message. Of course, this depends on me being responsible, and keeping my private key secure. We should now be familiar with all the concepts we’ll need to make sense of TUF. Earlier, we were concerned about whether an attacker might alter TUF metadata. Digital signatures are the solution. The TUF server treats the information (metadata) about the files we’re downloading as a message that it encrypts using its private key. Earlier, I left out that the Composer TUF plugin has a copy of the server’s public key. As a result, the plugin can decrypt the message, and thus be confident that it came from the TUF server. This means that the Composer TUF plugin can confirm the authenticity and integrity of the TUF metadata. Since it can trust the metadata, it can proceed to verify the files themselves. So a TUF server’s job is to: Generate metadata to allow a client to verify packages; Sign this metadata with private keys, so that the client can trust it; and Keep the private keys secure, so that attackers can’t forge metadata. Of course, this is all more complicated than what we’ve presented here. If you’re interested, I highly encourage you to read further on the TUF website As I mentioned at the very beginning, I’ve been working with the Drupal Association to build and deploy a TUF server to help secure downloads of modules and themes. This security infrastructure will become increasingly important as the Automatic Updates Initiative approches completion. One of the goals of this project has been to open-source the resulting code and documentation, to make it easier for other free/libre open source software (FLOSS) projects to adopt the TUF framework. We recently split out the TUF server component, that we’ve dubbed “Rugged”. Check it out at https://rugged.works https://consensus.enterprises/blog/salary-calculator/ Thu, 17 Mar 2022 09:00:00 -0400 Consensus Compensation Calculator on Consensus Enterprises Blog published Thu, 17 Mar 2022 09:00:00 -0400 When Consensus began, we were 4 equal partners doing roughly the same work, and generally equivalent levels of skill and experience. We all pitched in to run the company, while billing as many client hours as possible to keep the lights on. As we’ve grown, we’ve managed to find a balance between paying ourselves a fair wage and being able to cover expenses. Strategically, we’ve aimed to generate some profit with which to fund our longer-term goals, but not at the expense of a decent salary. … When Consensus began, we were 4 equal partners doing roughly the same work, and generally equivalent levels of skill and experience. We all pitched in to run the company, while billing as many client hours as possible to keep the lights on. As we’ve grown, we’ve managed to find a balance between paying ourselves a fair wage and being able to cover expenses. Strategically, we’ve aimed to generate some profit with which to fund our longer-term goals, but not at the expense of a decent salary. Above all else, we are humans first, with families to feed. We’ve always understood Consensus as a company that builds software products. We’ve had a few in mind from the very beginning, and more ideas have emerged along the way. We’ve also been committed to bootstrap our SaaS business line through client consulting, rather than seek venture capital or other such funding. This has presented us with some common problems, for which we’re experimenting with some uncommon solutions. Read on to find out how we’re thinking about a compensation model that’s genuinely equitable & transparent. From the start we recognized that we’d need to find an ongoing balance between the desire to increase our take-home pay, and the organizational goal to pay ourselves to build SaaS products. We built a simple paycheque calculator spreadsheet that took our expected schedule of working hours and translated it into paycheques. This soon morphed into serving various functions, one of which was to implement a “quarterly salary adjustment” process. This essentially formalizes the flexibility to set our own schedules, and allows us some agency over how much we take home. Once per quarter we review our timesheet reports, calculate how many hours we worked on a weekly basis, and then set our next quarter’s paycheques to correspond to that schedule. For example, suppose I work 35 hrs/wk consistently for the first quarter of the year. At the end of that 3 month period, we set my paycheque for the second quarter based on 35 hrs * $X/hr. Further suppose that in the second quarter, I have some family matters to attend to, and I average 30 hrs/wk. In that case, my third quarter paycheque would be calculated based on a 30hr week. We’ve found this approach has a number of benefits, but the key aspects of note here are flexibility and agency, from the perspective of a human worker with a full life outside of work. If I need to handle some personal business, there’s no need to ask for time off; if I want to increase my weekly schedule, there’s no need to adjust employment contracts, etc. Many times over the last 4 years, we have reflected on the challenges we’ve faced alongside the success we’ve enjoyed and remarked: “that’s a luxury problem, and a good one to have!” Too many clients to easily manage? Luxury Problem! Need to hire more people to cover all the projects we’re taking on? Luxury Problem! Found a whole other kind of role we can do? Luxury Problem! We’ve been blessed to find excellent clients and partners to work for and with. We believe we are also fairly talented at delivering high-quality, professional work, and have had many such clients come back asking for more. All of this has led to quickly growing our team over the last couple years. We are still fewer than 10, but we have doubled in that period, which in turn has stretched our compensation model to where it needed revisiting. When we first faced the challenge of hiring a new team member, we were lucky to find people much like ourselves, both in terms of expertise and competencies, but also in values and outlook. However, we recognized a disparity in the effort and risk we “founding partners” took on, starting a new company; this is commonly referred to as “sweat equity”. Whereas the partners might be working many overtime hours or unpaid to get a proposal delivered or some internal project complete, we could not expect the same of employees. Given this distinction, we felt it was reasonable to differentiate pay scale on that basis. As we’ve continued to grow, we’ve paid each new employee on par with everyone else, in an effort to have a rough equity at least. This was feasible at first, especially since we were hiring for very similar Drupal Developer roles. This became challenging for a couple of key reasons. First, we find ourselves trying to fill different roles, so there’s a need to differentiate on a finer scale based on skills, competencies, level of experience, and responsibility. Second, the weight of the founders’ sweat equity has decreased, and we needed a way to recognize the effort of our awesome new team members. Increasingly, the effort of the whole team keeping Consensus healthy is balancing out the initial bootstrap effort. In most functional terms, we collaborate as a team by consent-based decision-making. Which projects to pursue, what clients to take on, as well as organizational decisions, operational and strategic. Except where required for financial or legal reasons, the partners (as Directors of the Corporation), we strive to collaborate and find consensus in everything we do. Our salary calculations should reflect as much of the value each of us brings to the whole. (Related: we are also shifting our governance structure to address these exceptions.) Considering the above, and knowing what we do about pay equity (gender gaps intersected with racial gaps, along with socialized negotiation bias), we sought to define our compensation model in terms that suit our values. As much as possible, we want it to be: Transparent and accessible to all workers Fair and objective, applying measurable criteria to all workers Humane and respectful toward a diversity of workers Reflective of the breadth of skills and competencies we rely on in our work Flexible to change as needed, with market forces or internal dynamics Beyond seeking transparency (so everyone knows roughly what everyone else is making), and equity (so everyone knows they’re being paid the same for doing the same work), we also understand this as a way we can increase our individual agency as workers. Using an iterative process we will review and re-evaluate each person’s score on the various metrics which define the calculator. This ensures we regularly assess the stated goals, equalizing across similar roles and responsibilities. It also creates an opportunity to review the things Consensus values in our workers, and adjust our calculator to reflect them. Essentially, the salary calculator criteria become a guideline for professional development, providing advice on how to improve my wage. If I want to increase my value to the company, here are some concrete things I can work on, with objective metrics by which I can self-assess. Further, if I feel I am bringing value to the company outside of what’s captured in the current model, I have some agency here as well. I can engage in the review process whereby we re-evaluate and consider new factors to incorporate into the model. Of course, the devil is in the details, and we are in the process of hammering out the details of this process as we speak. We wanted to share some of our thinking to this point, primarily because our research has yet to reveal any prior art with the goals we have in mind. We’ll post more as we progress. Beyond this, we are seeking to build a model that can align our values as workers with the work we get paid for as a company. We aspire to get paid to do those things we are most passionate and excited about. We have noticed that asking potential recruits what they’re good at and what they’re excited about working on has been instructive. When we’ve been able to find projects that play to each person’s strengths, things go well! If we can establish that we like to work in similar ways on similar kinds of projects, the details of what we work on becomes somewhat less important. Have you tried to solve these problems? Do you know of some research we should consider in our process? Perhaps you’re excited about this idea, and want to talk to us about it? Get in touch! https://consensus.enterprises/blog/unicorns/ Fri, 25 Feb 2022 09:00:00 -0400 The Trouble with Unicorns on Consensus Enterprises Blog published Fri, 25 Feb 2022 09:00:00 -0400 When a small team knows that they need to get started in automation and they only have the budget for one (or maybe two) hires, they are on a unicorn hunt… they need a DevOps specialist who knows the particular programming languages they are currently using, who is an expert across a wide range of different technologies, who has enough experience to operate independently and can help with strategic direction and positioning, but is still cheap enough to work on an unfunded project, and also… … When a small team knows that they need to get started in automation and they only have the budget for one (or maybe two) hires, they are on a unicorn hunt… they need a DevOps specialist who knows the particular programming languages they are currently using, who is an expert across a wide range of different technologies, who has enough experience to operate independently and can help with strategic direction and positioning, but is still cheap enough to work on an unfunded project, and also… hey, can you fix that CSS issue on the front page while you are at it? If this is your problem, you don’t just need a unicorn, you need a specific unicorn. And there is no guarantee that they even exist, let alone that they exist in your network, and are currently looking for work. There are a few problems with this scenario. I’m not going to say unicorns don’t exist; every senior tech person I know has a unique combination of skills that, matched with the right job requirements, would make them look like a unicorn to the person hiring them. But: when you find yourself looking for a person with a precise combination of technical skills up and down the stack, merged with the ability to work with your existing group of clients and employees, now you’re looking for a very specific person. It’s like trying to find a perfect romantic partner… any potential magical unicorn you come across is almost certainly missing something vital that you need for this job. This brings us to: The previous issue can be summarized as such: The more circles you pile onto the Venn diagram, the fewer people exist at the intersection. It gets worse, though: you can’t change one of the circles and still have the same people in the middle. If you want to change the front end, the programming language, or the particular cloud architecture, you may have to start all over again, looking for a different unicorn. It’s a challenge to find all the right technical skills in one person. If you also need them to do the documentation, gather requirements from the client, run meetings, and keep all the moving parts of the project in mind to come in on time, you’re not looking for a unicorn any longer… you’re looking for a unicorn who is a mermaid (mer-person?) in their spare time. The more capabilities you need, the more expensive they get to hire. When you don’t need all their skills on an ongoing basis, you can wind up with a seriously overpowered staff member who is both costly, and at risk of getting bored if the maintenance bits are not as interesting as the ideation and architecture stage. In fact, a lot of the folks who are great at integration and strategy are not your best options for ongoing development and operations. Now you’ve got a pricey unicorn where you need a solid draft horse. And the last unicorn problem is… Here’s a secret… Full-Stack developers are mostly just OK at all the bits of the stack. The goal of hiring a full-stack developer is for them to know the end-to-end integration. They have to be good enough at each component to be able to respond to changes rapidly and as needed. They have to be aware of how all the pieces fit together so that they will know what their changes might impact up and down the line. However, they can’t possibly be an expert at all the components… and deep expertise is possible at every layer. Once you have moved beyond your team’s knowledge of the systems you are working with, especially if you are at the architecture stage, you’ll need that deep expertise, at least for a while. If you’re searching for a person to fill in the bits you don’t know, you’re playing a high-stakes game, because you don’t have any way to assess whether somebody else is good at things you don’t know how to do. Since you’re not an expert, you are not able to assess somebody else’s expertise, so you need to accept their claims of how things are/should be done. You don’t want to leave yourself in a position to be dazzled by somebody’s confidence when what you really need is somebody who can accurately identify and address risks. This is where you might consider engaging a team to get the next stage of your project built instead of hiring an employee right away. With a team, you get the benefits of high-level expertise without having to immediately identify and hire for a full-time role. You get the architect you need for a few months and the front-end developer without having to pick one or the other (or find one person who does both.) Now, let’s be honest, the hourly rate for a senior agency will come with a bit of sticker shock. You may have to pay more than you, personally, are getting paid, especially if you’re hiring on a short-term/low-hours contract. It’s more like what you pay your lawyer or your chiropractor than what you pay your employees. That can feel uncomfortable, but the benefit is getting access to a blend of the 1/4 of three different senior engineers that you actually need, instead of making do with an intermediate full-stack person (however wonderful) who has to spend a lot of their time trying to figure out the solution before implementing it. This is particularly relevant when backfilling temporary roles on a project. When you only need a short-to-mid-term commitment, you are in the enviable position of being able to hire skills, not people. As lovely as we are (and we really are), you’re not choosing somebody to work with for the next several years; you need somebody to build the missing parts of your system as fast and well as possible - to do the build, transfer knowledge to your long-term staff, and exit. It does not matter to your long-term success whether that skill set is spread over one or three people. It is just so much easier to fulfill your need for a particular intersection of skills with a group of ten senior developers than it is to find it all tidily bundled into one magical, full-stack, generalist/specialist unicorn. If you are currently at a stage where you need deep expertise in Automation or Architecture to get you through the next stage of your project, reach out to us below. We’d love to build a plan to get you on board with our team. https://consensus.enterprises/blog/sociocracy/ Fri, 28 Jan 2022 09:00:00 -0400 Aligned Governance for a Values-Driven Company on Consensus Enterprises Blog published Fri, 28 Jan 2022 09:00:00 -0400 I admit it. I’m here for the politics. I might be unique in this perspective, but in my not-so-humble opinion… Tech is tech. Languages are languages, frameworks are frameworks, and infrastructure is infrastructure. I can get any number of those at any number of places. I joined Consensus directly because of its positions on values, decision making, and the role of technology in society. We’re a non-standard bunch with a significant number of philosophy and social sciences courses (and degrees) … I admit it. I’m here for the politics. I might be unique in this perspective, but in my not-so-humble opinion… Tech is tech. Languages are languages, frameworks are frameworks, and infrastructure is infrastructure. I can get any number of those at any number of places. I joined Consensus directly because of its positions on values, decision making, and the role of technology in society. We’re a non-standard bunch with a significant number of philosophy and social sciences courses (and degrees) under our collective belts, so the conversation around the “water cooler” can get pretty deep. Because of our explicit commitment to liberatory and social justice principles, we also participate in the ongoing conversation about how to develop a new set of conventions around management - not only of our projects, but of the company as a whole. Coming from a background in agile development and open source software, we were looking for systems that allow space for autonomy, agency, and self-direction. We hire for the ability to think about things and solve problems… so we want to make sure that the people in the company get to perform at their best and make their best contributions. When I joined, I was the sixth non-contractor in the company. We were holding weekly all-hands coordination meetings to keep a handle on the moving parts. Even at this point, though, we were starting to run into the N^2 relationships problem. We were just at the beginning of growth, and we wanted to bake in good practices before we got too big to make changes. One of the most pressing problems was how to keep the right people informed about the right things without drowning everybody in minutiae. Every approach we tried seemed to be missing key components, and we were doing a lot of reinventing the wheel. As good academically-inclined folks, we knew that due-diligence involves research, so we went looking to see what other people were already doing. Fortunately for us, we don’t need to start from first principles. Since governance is an ongoing problem in organizations of all sizes, lots of smart people are working on these issues, and self-organization is an increasingly important approach. We wanted something that included autonomy, left much of the power in the hands of the people doing the work, but that also provided clarity of communication and direction, guardrails that could prevent costly errors that might undermine the viability of the company, and an approach that prevented silos. We needed information to flow with the power, so that decisions are made with maximal data, and we minimize surprises. Additionally, we need systems in place for all the normal issues of a for-profit company: We have to make good choices around budget and payroll, prevent over- or under-staffing without undermining the trust of our people (particularly by making sure there’s enough, but not too much, work to go around), and allow timely decision making. In the middle of this quest, one of our founders proposed sociocracy as a possible solution. It is related to holacracy, and is also called “Dynamic Governance.” To get started, we ordered copies of Ted Rau’s book on getting started with Sociocracy, “Who Decides Who Decides,” and the (somewhat heavier) “Many Voices, One Song.” (Ted Rau and Jerry Koch-Gonzalez) Now we had governance homework in between our coding sessions. The initial question was, “Is this worth pursuing?” After we had read enough to get the gist, we agreed that it seemed to check all the boxes, so we went as a group for foundational training. The first thing we noticed was that this is not an intuitive way of doing things. The facilitation processes in sociocracy formalize best practices around sharing the floor, sharing power, and sharing information. The free-form conversation that had become our habit leads to power imbalances, which is at odds with the goals we had just adopted. We needed to learn a new approach and a new set of skills. Sociocracy provides a complete structure and skill set that are designed to make sure that people get their concerns heard and considered. It incorporates a consent-based decision making process, with the goal of finding a next step that everybody can agree advances the aim of the group. That statement, though, sneaks in an assumption that you are clear about the aims. This is where a lot of the Hard Part arises in changing governance approaches: by the time an organization has been around for even a couple of years, everybody is carrying implicit assumptions about the purpose and goals. When you have to decide how to distribute resources (time, energy… money, especially) those implicit assumptions tend to come into conflict, often without an obvious cause. By adopting sociocracy, we committed to keep talking about the assumptions so that they become explicit. Sociocracy has given us a set of principles and practices to point back to as we make decisions and have the hard conversations. A few notes from our initial experience (after a handful of months): Even with some highly experienced facilitators in the company, getting the pace of the meetings right has been challenging. It is tempting to fall back into old habits, using a popcorn approach and letting the same people do most of the speaking. However, given current research on the importance of equal air time, making the effort to learn new patterns is worth it. If you skip rounds, check-ins, and meeting evaluations, you will miss out on vital information and you will lose the trust of the people who are being overlooked. Now, let’s acknowledge that you would have missed that by using a more traditional structure but in this case, you are also out of alignment, and it bothers people more. One of the things that can still trip us up adding an item to an agenda that looks like a one-off operational decision, but that turns out to have policy implications, either for future decisions, or for the company as a whole. These can come up unexpectedly; even the text in a public blog post can set expectations for clients and project partners, so you may think you’re just giving an update and find yourself enmeshed in a long-drawn-out discussion of everything that is planned for the next six months!!! This is commonly termed to be “No Good.” Our current solution is to review the agenda at the beginning of each meeting to identify items that may have hidden policy implications and take more time to decide… With each item, we need to ensure: Is this the right meeting? Is it clearly in the domain of this circle? Does anybody else need to be here? How long is this likely to take once we start to unpack it? Sometimes, we still miss these until we are in the weeds, but we’re getting better at catching them early. When we find we’ve tripped into the thorns, we prefer to find a set of questions that can be addressed, and then defer the larger conversation to another (possibly separate) meeting. We also have found that temporarily convened “helper circles” are useful for hashing out a rough solution which can then be considered by the larger group. We’re still running over on a lot of our meetings, but we’re improving. It takes a while to get good at this, and we’re concerned about putting too much of the governance skill into a small subset of the company. We’re trying to make sure that we all build our facilitation skills so that we can pick it up as necessary. Being in the facilitator’s role also increases everybody’s awareness of their own interactions, if they are jumping into the middle of rounds or doing a lot of cross-talk. When we have shared responsibility for keeping the meeting on track, we find that we are better at catching ourselves. If a meeting is running off the rails, it’s better to pause and get back on track than to carry on and try to get through the agenda. Similarly, the part at the end of meetings where we pause to consider the content, the process, and the interpersonal dynamics is important. Sometimes uncomfortable, but important. As a fledgling sociocracy organization, we’re still on the steep part of the learning curve. But having a formal structure that is so well aligned with our goals and operating principles has freed up a lot of our time that we were spending “reinventing the wheel.” We’ve also started to connect with the broader community around sociocracy, which reinvigorates our commitment to the underlying principles. We’re grateful to have found an approach that lets us treat governance as a largely “solved problem.” We are still learning, but not inventing, which frees us up to solve our technical and business problems. For more info: Sociocracy for All Brave New Work Many Voices, One Song Who Decides Who Decides https://consensus.enterprises/blog/virtual-water-cooler/ Wed, 24 Nov 2021 09:00:00 +0000 Remote Connections: Show Us All Your Pets! on Consensus Enterprises Blog published Wed, 24 Nov 2021 09:00:00 +0000 In the early days of Covid, the WSJ published an article about video etiquette that admonished us to, “Maintain professionalism: Don’t let your cats and children walk into the frame.” It garnered a range of disparaging responses, since it assumed that people had workspaces in their homes that were devoid of children and pets and that they magically had childcare available when none of us were allowed to leave our houses. Why didn’t the original author see that we were all doing the best we could … In the early days of Covid, the WSJ published an article about video etiquette that admonished us to, “Maintain professionalism: Don’t let your cats and children walk into the frame.” It garnered a range of disparaging responses, since it assumed that people had workspaces in their homes that were devoid of children and pets and that they magically had childcare available when none of us were allowed to leave our houses. Why didn’t the original author see that we were all doing the best we could under extremely strained circumstances? But there was another group of responses that centered on the humanizing aspects of seeing colleagues in their houses, realistically encumbered, with their cats and children climbing on them, and it was this: “Show us all your pets! Introduce us to your children! It’s the only thing that is making this situation bearable!” The second approach is the one we want to encourage in this post: not merely tolerating your colleagues’ humanity, but embracing it as a way to build connection and trust. As an entirely remote company, Consensus needs to provide ways to get to know one another artificially, because the conversations in the lunch room and around the water cooler don’t happen unless we make space for them. The company chat room doesn’t cut it. To acknowledge this need, we have coffee breaks on the calendar for the whole company, which are scheduled around people’s work days. These are the only meetings that we regularly turn on our cameras, and pets and children are definitely welcome. (In case you are wondering, the camera isn’t required; it’s just an option. We generally leave our cameras off for work meetings, both for privacy reasons and to reduce distractions on the screen. And at least one of our team members has no camera on their computer at all.) No matter how abstracted or technical our work becomes, we are still social creatures who rely upon reading one another’s faces and body language to know what’s happening in a conversation. When things go wrong in projects (even in projects that seem to be purely technical), it almost always comes down to questions of trust and communication. Regardless of how “meta” some people may want us to get, we still live in bodies in the physical world. When we go remote (especially by accident), we can be reduced to communicating through text or formal meetings. The stresses on conferencing systems and people’s home WiFi left a lot of people meeting purely by voice for the last year. And a lot of large companies (for good reasons) confined people’s communication to formal channels that could be monitored for inappropriate use. Unfortunately, the glue that binds a company together is informal communication: Gossip, if you must. We need to know what’s happening in people’s lives to make sense of what is happening in their days. If, for example, they are always required to pick up their children at 4PM, but we’re not allowed to know about that, all we see is that they keep disappearing for an hour in the afternoon. In the absence of information about people we don’t trust, we make up stories. (They’re often not very charitable.) Trust is not something that is given. It’s earned. For a project to succeed, there are some essential things that have to be true, that we also have to believe about one another. Most basically, the people working together have to honestly represent their skills, their progress, and their effort. Things go very wrong indeed when people start covering up challenges and failures or overpromising and underdelivering. At a higher level, we also have to trust that everybody on the project has a good outcome in mind. Ideally, there should be a clear overlap in those visions so that we are also working towards the same good outcome. Still more importantly, that outcome should benefit everybody on the team. We have probably all worked with people who didn’t meet these basic requirements of trust-worthiness… they were in it for themselves, they were actively undermining their coworkers, or they were in over their heads but were too embarrassed to admit it. But most of the places where it breaks down come down to “not knowing one another well enough.” This is a high risk for remote companies, and has been a huge focus of the debate about the Return to the Office: employers don’t trust their employees, and they haven’t come up with ways to bridge that gap other than through monitoring - which is what you use in a low-trust environment. It’s not an appropriate way to form and encourage high-performing teams. In the “when are you going back to the office” conversation, we’ve been left out. Like more and more digital companies, we never had an office to go to in the first place. But that doesn’t change the fundamentally social characteristics of trust. I want to point out some of the advantages of the remote workplace in getting to know one another. The blending of our work and home selves can have some significant boundary problems if we’re not careful. (9 pm text messages are right out.) But it can mean that we’re not covering things up in the same way. When one of our children is home sick, we all know about it. For that matter, when somebody’s babysitting falls through or they have to go to an appointment in the middle of the day, it’s much easier to say, “I’m not available,” when everybody understands more of the reality they are navigating. We recently had a long-delayed company retreat, and it was a bit odd to realize (as we walked up to one another in person)… that most of us had never actually “met” before. We know one another’s kids’ names and pets, and what kind of work their spouses do, what renovations are going on in their houses. We’ve seen the backdrops and piecemeal workspaces they resort to as they share the WiFi with varying numbers of family members. We’ve been “in” one another’s yards, and dining rooms, and hallways, and we’ve shared our various approaches to making coffee and the views out our windows in the middle of storms. (Generally with a caution that we may lose power at any moment.) We “know” one another in a way that we haven’t necessarily known our in-person colleagues in the past. A lot of the rhetoric around virtual spaces is that the relationships we form there aren’t “real.” To a certain extent, as long as they are mediated and kept artificially distant, that can be true. But if you want to build a cohesive remote team, you need to work around the limitations of the medium. It can be as simple as introducing your cats to one another. https://consensus.enterprises/blog/migrations-6-custom-table/ Tue, 14 Sep 2021 09:00:00 +0000 Easy commit credits with migrations, part 6: Migrating data from a custom table on Consensus Enterprises Blog published Tue, 14 Sep 2021 09:00:00 +0000 This is the sixth in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In part 2, I covered how to review and manually tested patches. In part 3, I demonstrated how to write automated migration tests In part 4, I demonstrated how to migrate simple configuration variables In part 5, I explained how to declare a module’s migration status to the migrate wizard In this part, I will show you how to migrate data from a custom database … This is the sixth in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In part 2, I covered how to review and manually tested patches. In part 3, I demonstrated how to write automated migration tests In part 4, I demonstrated how to migrate simple configuration variables In part 5, I explained how to declare a module’s migration status to the migrate wizard In this part, I will show you how to migrate data from a custom database table in D7. Stay tuned for more in this series! While migrating off Drupal 7 Core is very easy, there are still many contrib modules without any migrations. Any sites built using a low-code approach likely use a lot of contrib modules, and are likely blocked from migrating because of contrib. But — as of this writing — Drupal 7 still makes up 60% of all Drupal sites, and time is running out to migrate them! If we are to make Drupal the go-to technology for site builders, we need to remember that migrating contrib is part of the Site Builder experience too. If we make migrating easy, then fewer site builders will put off the upgrade or abandon Drupal. Plus, contributing to migrations gives us the opportunity to gain recognition in the Drupal community with contribution credits. In D7, some modules define their own database table to store data in (if the D7 module implemented hook_schema(), then there’s a pretty good chance it defines its own database table). In order to fully finish writing migrations for contrib modules that define custom tables, we need to know how to migrate data out of those custom tables. For example, Environment Indicator version 7.x-2.x stores data about each of its environments in a table named environment_indicator_environment, which has the following structure: Column name Type Options Notes machine varchar Length: 32, Unique constraint Environment ID name varchar Length: 255 Environment label envid serial Unsigned, Not null, Primary key Internal ID regexurl varchar Length: 255 Regular expression pattern to run on the URL to determine if a user is on the environment settings text Size: big, Serialized data … while the 4.x version of Environment Indicator (for D9) stores this data as configuration entities. We need to migrate data out of the D7 custom table and into the D9 config entities. You may be wondering “What is a configuration entity? I thought nodes, taxonomy terms and user accounts were entities?” Nodes, taxonomy terms, and/or user accounts group data with fields and are now called content entities. Similarly, a configuration entity in D9 groups configuration data with fields. But unlike content entities (whose IDs are usually numbers, e.g.: node 1, etc.), configuration entity IDs are usually machine names (e.g.: production, etc.). Also, while a content entity’s field names usually begin with field_; a configuration entity’s fields usually do not. Due to the similarities between content and configuration entities, they share a lot of code in D9; including the migrate destination plugin entity, which we will use. But what about migrating out of a custom table? The plugin in the list of core source classes that looks the closest to what we want to do is SqlBase… but SqlBase is marked as “abstract” (meaning that we cannot use it directly), because it’s query() function is abstract. In English, this means that the SqlBase class doesn’t know how to get the data that we want out of the custom table we want! We need to write our own custom Source plugin, which extends SqlBase, and implements its query() function. Let’s start by mapping the destination fields in the D9 configuration entity to the source fields in the D7 custom table, as we did for simple configuration: D9 field D9 field data type D9 field default value ← How to process ← D7 variable D7 data type Notes machine string (none) ← (copy) ← machine string (n/a) description text '' (empty string) ← (use default value) ← (doesn’t exist) (n/a) (n/a) name string (none) ← (copy) ← name string (n/a) (n/a) (n/a) (n/a) ← (discard) ← envid integer Required in D7 url uri (none) ← (copy) ← regexurl string (n/a) fg_color string #D0D0D0 ← (copy) ← settings.text_color string CSS color bg_color string #0D0D0D ← (copy) ← settings.color string CSS color (n/a) (n/a) (n/a) ← (discard) ← settings.weight (doesn’t matter) (n/a) (n/a) (n/a) (n/a) ← (discard) ← settings.position (doesn’t matter) (n/a) (n/a) (n/a) (n/a) ← (discard) ← settings.fixed (doesn’t matter) (n/a) Given this information, we can write a migration configuration at migrations/d7_environment_indicator_hostname_environments.yml: id: d7_environment_indicator_hostname_environments label: Environment indicator hostname environments migration_tags: - Drupal 7 - Configuration source: plugin: d7_environment_indicator_hostname_environment process: machine: machine name: name url: regexurl fg_color: text_color bg_color: color description: - plugin: default_value default_value: '' destination: plugin: 'entity:environment_indicator' Note the custom source plugin, d7_environment_indicator_hostname_environment: this plugin doesn’t exist yet — we will write it shortly. The migration’s process configuration should look familiar: we simply map source variables to destination variables; although you’ll notice we’re leaving out the settings. prefix that we used in the mapping table for D7’s settings.text_color and settings.color — because we are writing our own source plugin, we can name the data fields whatever we want. Finally, we specify the destination plugin entity:environment_indicator. This is the entity migration destination plugin that we mentioned earlier; plus the destination entity ID environment_indicator. We get this ID from the entity type that we’re migrating into — in this case, the configuration entity defined in environment_indicator-4.x’s Drupal\environment_indicator\Entity\EnvironmentIndicator class. Before we get too much further; we should write a test for the migration we just wrote. In tests/src/Kernel/Migrate/d7/MigrateHostnameEnvironmentsTest.php: namespace Drupal\Tests\environment_indicator\Kernel\Migrate\d7; use Drupal\Core\Database\Database; use Drupal\environment_indicator\Entity\EnvironmentIndicator; use Drupal\Tests\migrate_drupal\Kernel\d7\MigrateDrupal7TestBase; /** * Tests migration of environment_indicator hostname environments. * * @group environment_indicator */ class MigrateHostnameEnvironmentsTest extends MigrateDrupal7TestBase { /** {@inheritdoc} */ protected static $modules = ['environment_indicator']; /** {@inheritdoc} */ protected function setUp(): void { parent::setUp(); // Create the environment_indicator_environment table in the D7 database. // The schema definition here was copied from version 7.x-2.9. Database::getConnection('default', 'migrate') -schema() -createTable('environment_indicator_environment', [ 'fields' = [ 'machine' = [ 'type' = 'varchar', 'length' = '32', 'description' = 'Unique ID for environments.', ], 'name' = [ 'type' = 'varchar', 'length' = '255', 'description' = 'Name for the environments.', ], 'envid' = [ 'type' = 'serial', 'unsigned' = TRUE, 'not null' = TRUE, 'description' = 'Primary ID field for the table. Not used for anything except internal lookups.', 'no export' = TRUE, ], 'regexurl' = [ 'type' = 'varchar', 'length' = '255', 'description' = 'A regular expression to match against the url.', ], 'settings' = [ 'type' = 'text', 'size' = 'big', 'serialize' = TRUE, 'description' = 'Serialized array with the configuration for the environment.', ], ], 'primary key' = ['envid'], 'unique keys' = [ 'name' = ['machine'], ], ]); $this-setUpD7EnableExtension('module', 'environment_indicator', 7202, 0); } /** Simulate enabling an extension in the D7 database. */ protected function setUpD7EnableExtension($type, $extensionName, $schemaVersion, $weight) { $extensionName = strval($extensionName); Database::getConnection('default', 'migrate') -upsert('system') -key('name') -fields([ 'filename', 'name', 'type', 'owner', 'status', 'bootstrap', 'schema_version', 'weight', ]) -values([ 'filename' = sprintf('sites/all/modules/%s/%s.module', $extensionName, $extensionName), 'name' = $extensionName, 'type' = strval($type), 'owner' = '', 'status' = 1, 'bootstrap' = 0, 'schema_version' = intval($schemaVersion), 'weight' = intval($weight), ]) -execute(); } /** Tests migrating hostname environments. */ public function testHostnameEnvironmentMigration() { // Fixtures we can verify. $machine = 'production'; $name = 'Production'; $url = 'https://prod.example.com'; $bgColor = '#abc123'; $fgColor = '#def456'; // Fixtures that won't be migrated. $envid = 17; // Set up the D7 environment and run the migration. $this-setUpD7HostnameEnvironment($machine, $name, $envid, $url, $bgColor, $fgColor, 11, 'top', FALSE); $this-executeMigrations(['d7_environment_indicator_hostname_environments']); // Load the D8 environment indicator and verify against the fixtures. $env = EnvironmentIndicator::load($machine); $this-assertInstanceOf('Drupal\environment_indicator\Entity\EnvironmentIndicator', $env); $this-assertSame($env-label(), $name); $this-assertSame($env-getUrl(), $url); $this-assertSame($env-getBgColor(), $bgColor); $this-assertSame($env-getFgColor(), $fgColor); } /** Add a D7 hostname environment to be migrated. */ protected function setUpD7HostnameEnvironment($machine, $name, $envid, $regexUrl, $bgColor, $textColor, $weight, $position, $fixed) { $this-assertIsString($machine, 'Machine name must be a string.'); $this-assertIsString($name, 'Name must be a string.'); $this-assertIsInt($envid, 'Envid must be an integer.'); $this-assertIsString($regexUrl, 'RegexURL must be a string.'); $settings = [ 'color' = strval($bgColor), 'text_color' = strval($textColor), 'weight' = strval($weight), 'position' = strval($position), 'fixed' = boolval($fixed), ]; Database::getConnection('default', 'migrate') -upsert('environment_indicator_environment') -key('machine') -fields(['machine', 'name', 'envid', 'regexurl', 'settings']) -values([ 'machine' = $machine, 'name' = $name, 'envid' = $envid, 'regexurl' = $regexUrl, 'settings' = serialize($settings), ]) -execute(); } } You’ll notice a few new things in this test: firstly, we use Drupal’s Upsert query to insert or update a row in the database — using an upsert here makes sure that the row we’re testing matches what we expect, without having to check whether a row with the same key already exists. Although we’re not doing it here for clarity; this is useful when randomizing test fixture data. The equivalent raw SQL for Drupal’s Upsert varies based on your database backend — it becomes INSERT ... ON DUPLICATE KEY UPDATE in MySQL; and INSERT ... ON CONFLICT (...) DO UPDATE in PostgreSQL and SQLite. Also new in this test is a setUp() function — if you recall from PHPUnit’s documentation on writing tests (which was extra reading in part 3 of this blog series), setUp() is run before every test function. In this case, we use it to set up a database schema which we copied from the 7.x-2.9 version of Environment Indicator, i.e.: the version that we’re migrating from. We also have to give the D7 module a row in D7’s system table (which we do in our test’s setUpD7EnableExtension() function). The test itself (testHostnameEnvironmentMigration()) should look pretty familiar by now: we set up fixtures (using the setUpD7HostnameEnvironment() function to clean things up a bit), run the migration, and test that the fixture data was migrated to the destinations that we expected. Finally, we need to write a custom source plugin… in src/Plugin/migrate/source/d7/D7HostnameEnvironment.php: namespace Drupal\environment_indicator\Plugin\migrate\source\d7; use Drupal\migrate\Row; use Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase; /** * Drupal 7 Environment Indicator Hostname Environment source from database. * * @MigrateSource( * id = "d7_environment_indicator_hostname_environment", * source_module = "environment_indicator" * ) */ class D7HostnameEnvironment extends DrupalSqlBase { /** {@inheritdoc} */ public function fields() { return [ 'machine' = $this-t('Unique ID for environments.'), 'name' = $this-t('Name for the environments.'), 'envid' = $this-t('Primary ID field for the table. Not used for anything except internal lookups.'), 'regexurl' = $this-t('A regular expression to match against the url.'), 'settings' = $this-t('Serialized array with the configuration for the environment.'), 'text_color' = $this-t('The text color of the environment indicator.'), 'color' = $this-t('The background color of the environment indicator.'), ]; } /** {@inheritdoc} */ public function getIds() { $ids['envid']['type'] = 'integer'; return $ids; } /** {@inheritdoc} */ public function query() { return $this-select('environment_indicator_environment', 'eie') -fields('eie', [ 'machine', 'name', 'envid', 'regexurl', 'settings', ]); } /** {@inheritdoc} */ public function prepareRow(Row $row) { $settings = unserialize($row-getSourceProperty('settings')); $row-setSourceProperty('text_color', $settings['text_color']); $row-setSourceProperty('color', $settings['color']); return parent::prepareRow($row); } } We declare the source plugin ID in the @MigrateSource annotation — this has to match the source plugin ID that we reference in the migration (migrations/d7_environment_indicator_hostname_environments.yml). Our custom source plugin class extends DrupalSqlBase, which in turn extends the SqlBase class we found earlier when we were looking for source plugins (DrupalSqlBase adds a few Drupal-specific checks and logic). In the fields() function, we declare which data fields we are going to pass from this custom source plugin to the process part of the migration (this is where we declare text_color and color without the settings prefix). We declare settings and envid even though our migration doesn’t use them, because we need to handle these fields internally in this class. In the getIds() function, we return the field envid and its type. Drupal 9’s migration subsystem uses fields that you declare in getIds() to understand which data has been migrated, and which data still needs to be migrated (the data returned by this function is used when rolling-back and resuming migrations). In the query() function, we return a simple Drupal Select query to get data out of the D7 table. The prepareRow() function runs on each result from the query declared in query(). Here, we perform some post-processing, in this case, by unserializing the data in the settings column, and using it to populate the (unprefixed) text_color and color fields we declared in fields(). When you’re writing a custom source plugin, it might be a good idea to make all of the D7 data fields available (provided that it doesn’t add too much additional complexity): another module might extend the one you’re working on, and might want to migrate data from the fields you decided to ignore. For a more complete example, check out my patch to migrate configuration from environment_manager, which also includes a custom process plugin. Starting next week, I’ll be taking a bit of a break from this blog series — new posts won’t be coming out as often — but I hope to eventually explore custom process and destination plugins, how to migrate content, and how to migrate data from custom field types defined by a module. https://consensus.enterprises/blog/migrations-5-migration-status/ Tue, 07 Sep 2021 09:00:00 +0000 Easy commit credits with migrations, part 5: Declaring a module's migration status on Consensus Enterprises Blog published Tue, 07 Sep 2021 09:00:00 +0000 This is the fifth in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In part 2, I covered how to review and manually tested patches. In part 3, I demonstrated how to write automated migration tests In part 4, I demonstrated how to migrate simple configuration variables In this part, I’ll show you how to declare a contrib module’s migration status, i.e.: to indicate whether you’ve written all the migrations that you intended to … This is the fifth in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In part 2, I covered how to review and manually tested patches. In part 3, I demonstrated how to write automated migration tests In part 4, I demonstrated how to migrate simple configuration variables In this part, I’ll show you how to declare a contrib module’s migration status, i.e.: to indicate whether you’ve written all the migrations that you intended to write. In part 6, I show how to migrate data from a custom database table. Stay tuned for more in this series! While migrating off Drupal 7 Core is very easy, there are still many contrib modules without any migrations. Any sites built using a low-code approach likely use a lot of contrib modules, and are likely blocked from migrating because of contrib. But — as of this writing — Drupal 7 still makes up 60% of all Drupal sites, and time is running out to migrate them! If we are to make Drupal the go-to technology for site builders, we need to remember that migrating contrib is part of the Site Builder experience too. If we make migrating easy, then fewer site builders will put off the upgrade or abandon Drupal. Plus, contributing to migrations gives us the opportunity to gain recognition in the Drupal community with contribution credits. In the last blog post, we walked through the process of creating a simple configuration migration — but I noted that, even after you’ve built the migration, when you get to the “What will be upgraded?” step in the migration wizard, the module will still show up in the list of “Modules that will not be upgraded”. This happens because core’s Migrate Drupal UI has no way of knowing whether you’ve written all the migrations that you intended to write! If you look closely at the “What will be upgraded?” step, you’ll see there is a row for each module that has stored data on the D7 site — that is to say, D7 modules which do not store data are not listed; and D9 modules are only mentioned if they declare a migration for the data in one of those D7 modules. Also, to date, this blog series has assumed that you are migrating to D9 from an older D7 version of the same module — but that doesn’t necessarily need to be the case: for example, the D9 Address module didn’t exist in D7: its predecessor module was named AddressField. Address module migrations would be written to migrate data from the AddressField module. Recall that the main goal of this blog series is to improve the upgrade experience for Site Builders… as a Site Builder facing an upgrade, I want as many of my D7 modules to be (accurately) accounted for in the “What will be upgraded?” step of the migration wizard, so that I know how much manual migration that I need to do after running the migration wizard. In Drupal 8.8, the migration team introduced a way for modules to declare their upgrade status. The status determines whether the “What will be upgraded?” report will list a D7 module in the list of “Module(s) that will be upgraded” or “Module(s) that will not be upgraded”. A migration status looks like… # In migrations/state/D9_DESTINATION_MODULE.migrate_drupal.yml finished: 6: d6_source_module_1: D9_DESTINATION_MODULE 7: d7_source_module_2: D9_DESTINATION_MODULE d7_source_module_3: - D9_DESTINATION_MODULE - other_d9_destination_module not_finished: 7: d7_source_module_4: D9_DESTINATION_MODULE You can see from this example that: You declare migrations as either finished or not_finished. In the “What will be upgraded?” report, a source module that does not have a migration declared for it — or whose migration is declared as not_finished — will appear in the “Module(s) that will not be upgraded” list. If a migration is declared as finished, then the module will appear in the “Module(s) that will be upgraded” list. You declare migration statuses for D6 and D7 modules separately. This allows you to tackle D6 and D7 migrations separately. You can declare migrations from one or more source modules to one or more destination modules. For example, core’s telephone module declares that it can migrate content from both the D7 Phone module and the the D7 Telephone module. Unfortunately, I’m not aware of an example where more than one D9 destination module is defined for a D7 source module. You declare migrations as finished or not_finished for the module as a whole. For example, this means that if a D7 module stores both content AND configuration, and you’ve only written a migration for configuration, then the module’s status is not_finished. Only once you’ve written the migration for the content, you can declare the status as finished. Let’s try to follow the principles of test driven development (TDD) by writing a test before we write the code to make that test pass. Put the following template at your module’s tests/src/Kernel/Migrate/d7/ValidateMigrationStateTest.php: namespace Drupal\Tests\MODULE_NAME\Kernel\Migrate\d7; use Drupal\Tests\migrate_drupal\Kernel\d7\MigrateDrupal7TestBase; use Drupal\Tests\migrate_drupal\Traits\ValidateMigrationStateTestTrait; /** * Tests that the MODULE_NAME test has a declared migration status. * * ValidateMigrationStateTestTrait::testMigrationState() will succeed if the * modules enabled in \Drupal\Tests\KernelTestBase::bootKernel() have a valid * migration status (i.e.: finished or not_finished); but will fail if they do * not have a declared migration status. * * @group MODULE_NAME */ class ValidateMigrationStateTest extends MigrateDrupal7TestBase { use ValidateMigrationStateTestTrait; /** * {@inheritdoc} */ protected static $modules = ['MODULE_NAME']; } The test inherits from MigrateDrupal7TestBase, which automatically sets up a migration; and the test includes code fromValidateMigrationStateTestTrait — which has a public function testMigrationState() — so the migration state is automatically tested if you just fill in the MODULE_NAME. Since we haven’t declared a state yet, if you run this test, it will fail. Now, let’s write a migration state! At migrations/state/MODULE_NAME.migrate_drupal.yml… not_finished: 7: D7_MODULE_NAME: D9_MODULE_NAME Now, when you run the test, it will pass, because the module has declared a status (even though the status is not_finished). Once you’re confident that you’ve written migrations for all the data that your D7 module can store, you can change that not_finished to finished. If you’ve already contributed some migrations, you can update those contributions to declare the migration status for that module. Remember, only declare the status as finished if you’ve written migrations for all the data stored by the D7 module that can be stored by the D9 module. https://consensus.enterprises/blog/migrations-4-simple-config/ Tue, 31 Aug 2021 09:00:00 +0000 Easy commit credits with migrations, part 4: Migrating D7 variables on Consensus Enterprises Blog published Tue, 31 Aug 2021 09:00:00 +0000 This is the fourth in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In part 2, I covered how to review and manually tested patches. In part 3, I demonstrated how to write automated migration tests In this part, I’ll demonstrate how to migrate simple configuration (“variables” in D6 and D7 parlance). In part 5, I explain how to declare a module’s migration status to the migrate wizard. In part 6, I show how to migrate data from a … This is the fourth in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In part 2, I covered how to review and manually tested patches. In part 3, I demonstrated how to write automated migration tests In this part, I’ll demonstrate how to migrate simple configuration (“variables” in D6 and D7 parlance). In part 5, I explain how to declare a module’s migration status to the migrate wizard. In part 6, I show how to migrate data from a custom database table. Stay tuned for more in this series! While migrating off Drupal 7 Core is very easy, there are still many contrib modules without any migrations. Any sites built using a low-code approach likely use a lot of contrib modules, and are likely blocked from migrating because of contrib. But — as of this writing — Drupal 7 still makes up 60% of all Drupal sites, and time is running out to migrate them! If we are to make Drupal the go-to technology for site builders, we need to remember that migrating contrib is part of the Site Builder experience too. If we make migrating easy, then fewer site builders will put off the upgrade or abandon Drupal. Plus, contributing to migrations gives us the opportunity to gain recognition in the Drupal community with contribution credits. In my experience as a consultant, clients who are willing to be early adopters of Drupal 7 to 9 migrations tend to want to make a bunch of other changes to their site at the same time… so configuration has often been overlooked in favour of setting new config from scratch on the D9 site. But from an end-user-of-Drupal’s standpoint, when the budget is tight, and/or Drupal 7 already functions the way you want it, it makes more sense to spend your time and money on verifying the site’s content, and updating the website’s theme! As a Site Builder migrating a site from Drupal 7 to Drupal 9, I want as much of my Drupal 7 configuration to be migrated as possible, so that I can spend my time on the theme and content of the site. Define a migration for simple configuration from Drupal 7 to Drupal 9. As mentioned briefly in the last post, migrations are defined by YAML files inside a module’s migrations/ directory that look something like… id: MIGRATION_NAME label: A Human-Friendly Name migration tags: - Drupal 7 - A Migration Tag source: plugin: # a @MigrateSource plugin id # some config for that @MigrateSource plugin process: # some process config destination: plugin: # a @MigrateDestination plugin id # some config for that @MigrateDestination plugin As you can probably guess, id, label, and migration tags are metadata. Each migration definition includes a source plugin and its configuration, which states where to find data in the Drupal 7 source database. Each migration also defines a destination plugin and its configuration, which tells Drupal 9 where to store the migrated data. Each migration also contains a number of process instructions, which describe how to build the destination data, usually by taking data out of the source. Before we can write a simple config migration, we need to understand how config is stored in both systems; and do a bit of planning. In Drupal 9, the standard way to handle configuration is to store it as configuration entities, using the configuration management API. Most configuration migrations into D9 use the config destination plugin, which uses the configuration management API to write the data. You configure the config destination plugin by specifying the machine name of the configuration entity that you want to build from the migrated data. If we look at the example migration we wrote tests for in the last blog post (i.e.: migrating config for the Environment Indicator module), you can see in its destination section… destination: plugin: config config_name: environment_indicator.settings … that the migration is going to be building the config object named environment_indicator.settings. Note that each migration has one destination plugin; and the config destination plugin only lets you specify one config entity. To start a configuration migration, I usually look at the Drupal 9 module’s code for configuration objects. If there is more than one, I start with the one containing general configuration settings. Once I’ve chosen a destination configuration object to focus on, I look at its definition in the module’s config/schema/MODULE_NAME.schema.yml file and where the config is being used (because the schema file isn’t always kept up-to-date). I start an inventory of the fields in that config object, their data type, and their default values from config/install/*.yml. A spreadsheet is a great tool for this inventory (just be aware that it can be helpful to show the spreadsheet to the community). In Drupal 7, the standard way to handle configuration was to store it in Drupal 7’s variable table in the database; and interact with it using the variable_get(), variable_set() and variable_del() functions. My next step in writing a configuration migration is to search the D7 module’s code for the string variable_, examine the different variable names, and update my inventory with the D7 variable names and data types (to determine a variable’s data type, you may have to look at how the D7 code uses it). Some modules construct their variable names by concatenating strings and variables, so one string match (for variable_) may correspond with a handful of possible variables. If the way that variable names are constructed is particularly convoluted, it can be helpful to install the module on your D7 site, configure it, and see which variables are added to the variable table in the database. When writing our migration definition for Drupal 7 variables, we can use the variable source plugin to pull data from the D7 variables table. You configure the variable source plugin by specifying a bunch of variable names to read data from; and optionally, specify which D7 module you’re migrating from (which is useful when you’re migrating config from a bunch of D7 modules into one D9 module). If we look at the example migration we wrote tests for in the last blog post, you can see in its source section… source: plugin: variable variables: - environment_indicator_integration - environment_indicator_favicon_overlay source_module: environment_indicator … that the migration is going to copy data out of the environment_indicator_integration and environment_indicator_favicon_overlay variables. At this point, your inventory should contain the names, data-types, and default values for a bunch of D9 config object fields; plus the names and data-types for a bunch of D7 variables. The next step is to process the inventory: for each D9 config object field, see if you can find a corresponding D7 variable, and mark the relationship in the inventory. It is always worth comparing how a config variable is used in both versions of the module, just in case it is unrelated but happened to be given a similar name. You should expect to find D7 config which does not have corresponding config in D9 and vice-versa. If you see D9 config that is related to the D7 config, but isn’t an exact copy (e.g.: a single value in D7 is the first value in an array in D9), add a note… we’ll talk about this shortly. When you are done, your inventory might look like this… D9 field D9 field data type D9 field default value ← How to process ← D7 variable D7 data type Notes toolbar_integration array [] ← (copy) ← environment_indicator_integration array (n/a) favicon boolean FALSE ← (copy) ← environment_indicator_favicon_overlay boolean (n/a) At this point, you have enough information to start writing the migration test, which we covered in the previous blog post. Now that we know how to migrate the data, we can write the process part of the migration configuration. Each instruction in the process section is a mapping (i.e.: hash, dictionary, object) whose name is the destination field. Inside the mapping is a list of migrate process plugins, to be run in order from first to last. The value after the final process plugin has run gets inserted into the destination field. To get data from the source, you use the get migrate process plugin, which you configure by specifying which source fields to use… process: toolbar_integration: # i.e.: the destination field in the 'environment_indicator.settings' config object - plugin: get source: - environment_indicator_integration # i.e.: the source field favicon: - plugin: get source: - environment_indicator_favicon_overlay … this configuration copies the data in the environment_indicator_favicon_overlay variable from D7, performs no other processing on it (i.e.: because there are no other instructions), and inserts it into the favicon field in the environment_indicator.settings config object. Since copying data from a source field to a destination field without any processing is so common, there is a short-hand for this particular case. For example, process: favicon: - plugin: get source: - environment_indicator_favicon_overlay … is equivalent to… process: favicon: environment_indicator_favicon_overlay After converting the plugin: get lines to the shorthand, your migration config should look identical to the sample one that I gave in the previous blog post. If you replace migration given last time, with the one you just finished building (don’t forget to set the migration id — it must match the filename and be in the $this-executeMigrations(['...']); line in your test). You can verify this is the case by running the test again. If you do need to modify the D7 data before it gets saved to D9, it is possible to add additional process plugins. For example, the following configuration would get the data stored in the source’s my_d7_label field, URL-encode it, convert that URL-encoded data to a machine name, then store the resulting data to dest_field… process: dest_field: - plugin: get source: - my_d7_label - plugin: urlencode - plugin: machine_name … put another way, given the data A name in source field my_d7_label, the data written to destination field dest_field would be a_20name (i.e.: A name - A%20name - a_20name). If you are performing other processing steps, Core and many Contrib process plugins will allow you to take a shortcut by replacing the stand-alone get step by specifying a source in the first processing step. For example, process: dest_field: - plugin: get source: - my_d7_label - plugin: urlencode - plugin: machine_name … is equivalent to… process: dest_field: - plugin: urlencode source: my_d7_label - plugin: machine_name … but you may find it easier to keep the stand-alone get step until you’ve completed the whole migration and you are certain that it works. When you write tests for a migration that involves process steps which modify data, the data you add in your test fixtures will be different from the data you verify at the end of the test — explicitly calling this out in a comment can be helpful to other people reading the test (or yourself 6 months later, when you’ve forgotten why). Once a Drupal 7 module has been ported to D9 for the first time, that module’s D7 and D9 codebases diverge. Features added to the D9 version aren’t always backported to the D7 version for various reasons. As a result, when preparing your inventory, it’s not unusual to find that the D9 config object has fields for configuration which doesn’t exist in D7 (note the converse — where D7 variables have no D9 equivalent — is possible too, albeit less common). When you’re writing a migration for the first time, it’s easy to focus on the config that you can migrate from D7, and ignore the D9 config which has no D7 equivalent. But if you don’t specify a value for those fields, the config destination plugin will set them to NULL when it constructs the config object from the migrated data. But, many modules assume that those configuration object fields will be set to their default configuration (i.e.: from config/install/*.yml) — not NULL — which can lead to bugs, errors, warnings, and crashes later on. The solution is to specify default values for that configuration in the process section of the migration definition, using the default_value process plugin. For example: process: d9_only_feature: - plugin: default_value default_value: 'some_default_value' When you specify default values, you should still test them, by verifying them when you verify the migrated data. I tend to separate these into their own section with a comment, so that I don’t get confused about why those verification lines don’t have a corresponding fixture… // Verify the fixtures data is now present in the destination site. $this-assertSame(['toolbar' = 'toolbar'], $this-config('environment_indicator.settings')-get('toolbar_integration')); $this-assertSame(TRUE, $this-config('environment_indicator.settings')-get('favicon')); // Verify the settings with no source-site equivalent are set to their default values in the destination site. $this-assertSame('some_default_value', $this-config('environment_indicator.settings')-get('d9_only_feature'); Make sure your Drupal 9 environment is set up as described in the last blog post: Clone Drupal core, run composer install, set up the site. Find a migration issue and module to work on. Clone the module to modules/, and switch to the branch in the migration issue’s Version field if necessary. If the migration issue is using an issue fork, then switch to the issue fork using the instructions in the issue. If the migration issue is using patches, download the patch, apply it to a branch named after the issue ID and comment number the patch was uploaded to (we’ll call this $FIRST_BRANCH below). If the migration issue is using patches, then create a second branch named after the issue ID and number of comments in the issue plus 1; and apply the patch, but don’t commit it yet. Create the migration inventory. Write the migration test. Write the migration itself, running tests frequently. Spin up your D7 site, install the D7 version of the module, and run a manual test, as we did in part 1 of this series. The automated tests only test for very specific problems on a simulated “clean” environment — which make them great for catching regressions — but not very good for catching problems you weren’t specifically testing for (that is to say, things likely to crop up in the real world). Note that Drupal 9 core’s Migrate Drupal UI module has no way of knowing if you’ve written all the migrations that you intended to write for this module. So, the module you’ve written the migration for will still show up in the list of “Modules that will not be upgraded” for now — we’ll fix that in the next blog post. Don’t worry though, your migration will still run. When you’re satisfied, stage all the changes to the module (i.e.: git add .), and commit your changes. In the commit message, describe what you did. The commit message will be visible to other members of the community. If the migration issue is using an issue fork, then push your changes to the issue fork, and leave a comment in the issue describing what you did. If the migration issue is using patches, then: Generate the patch with git format-patch 8.x-2.x (where 8.x-2.x is the branch specified in the “Version” field of the issue with the patch). Generating a patch in this way way adds some metadata which will help avoid merge conflicts in the future. The patch will appear in the current directory, and will be named something like 0001-YOUR-COMMIT-MESSAGE.patch. Rename the patch according to Drupal.org’s conventions for naming patches. I like to move the patch somewhere that I can easily find it (e.g.: my Desktop folder) at the same time that I’m renaming it. Generate an interdiff between your current patch and the previous one with git diff $FIRST_BRANCH interdiff.txt (where $FIRST_BRANCH is the branch with the previous patch applied and committed). I like to move the interdiff somewhere that I can easily find it (e.g.: my Desktop folder). Start a new comment, upload the patch and interdiff, and describe what you did in the Comment. If you set the issue status to “Needs review”, then automated tests will run on your patch — but once they pass, change the issue status back to “Needs work”, because your migration won’t be finished until you’ve verified there’s nothing else to migrate, and indicated that to the Migrate Drupal UI module. If you’ve been following along with our example to migrate configuration for the Environment Indicator module, please be aware that there’s already a migration to do that in issue #3198995 — so please do not create a new issue, and please do not leave patches in that issue. At this point, you should have all the tools that you need to contribute patches which migrate simple configuration from the Drupal 7 version of a module to the Drupal 9 version of the module, so try it out! Next, we will talk about how to tell Drupal core’s Migrate Drupal UI module that you’ve written all the migrations that you intended to write, which will move the module from “Modules that will not be upgraded” to “Modules that will be upgraded” in the migration wizard. We’ll also talk about how to migrate more complex configuration and content in future posts in this series. https://consensus.enterprises/blog/migrations-3-automated-testing/ Tue, 24 Aug 2021 09:00:00 +0000 Easy commit credits with migrations, part 3: Automated tests on Consensus Enterprises Blog published Tue, 24 Aug 2021 09:00:00 +0000 This is the third in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In part 2, I covered how to review and manually test patches. In this part, I’ll demonstrate how to write automated migration tests. In part 4, I discuss migrating simple configuration. In part 5, I explain how to declare a module’s migration status to the migrate wizard. In part 6, I show how to migrate data from a custom database table. Stay tuned for more in … This is the third in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In part 2, I covered how to review and manually test patches. In this part, I’ll demonstrate how to write automated migration tests. In part 4, I discuss migrating simple configuration. In part 5, I explain how to declare a module’s migration status to the migrate wizard. In part 6, I show how to migrate data from a custom database table. Stay tuned for more in this series! While migrating off Drupal 7 Core is very easy, there are still many contrib modules without any migrations. Any sites built using a low-code approach likely use a lot of contrib modules, and are likely blocked from migrating because of contrib. But — as of this writing — Drupal 7 still makes up 60% of all Drupal sites, and time is running out to migrate them! If we are to make Drupal the go-to technology for site builders, we need to remember that migrating contrib is part of the Site Builder experience too. If we make migrating easy, then fewer site builders will put off the upgrade or abandon Drupal. Plus, contributing to migrations gives us the opportunity to gain recognition in the Drupal community with contribution credits. In the last blog post, we tested migration patches by manually creating content in the D7 site, and manually verifying that the content we created was migrated to the new site. But entering test data, running the migration, and verifying the test data by hand is tedious and error-prone, especially if we want to be able to perform the exact same tests a few months later to ensure that recent changes to the module haven’t caused a regression by breaking the migration! Being able to quickly run a migration is also quite useful when writing a migration from scratch (a topic we will cover in future blog posts), because you can get continuous feedback on whether your changes were effective (i.e.: you can do test driven development (TDD) — a style of programming where you (1) write a (failing) test, (2) write operational code so the test passes, and (3) refactor… and you repeat that cycle until you’ve solved the problem). Let’s automate running the migration: automation will ensure that the test is performed the same way next time. We will do so by writing PHPUnit tests. PHPUnit is an automated testing tool used in Drupal core. Because Drupal’s PHPUnit tests run in an isolated environment, this will save us time reverting the database before each migration test. As an added bonus, Drupal CI — Drupal.org’s testing infrastructure — can be configured to run tests when patches and/or merge requests are posted to the module’s issue queue, to remind other contributors if the change they are proposing would break migrations in some way. Migration tests typically follow a pattern: Set up the migration source database, Fill the migration source database with data to migrate (“set up Test Fixtures”), Run the migration (“run the System Under Test”), and, Verify the migration destination database to see if the test fixtures were migrated successfully. You might notice that we’ve been following this pattern in our manual tests. PHPUnit tests themselves are expressed as PHP code. Note that this is different from Behat behavioural tests (where tests are expressed in the Gherkin language), or visual regression tests (where — depending on your testing tool — tests could be expressed as JavaScript code, as a list of URLs to compare, etc.). Drupal’s convention is to put D7 migration tests into a module’s tests/src/Kernel/Migrate/d7/ folder. You’ll find many Core modules with migration tests in this location (Core’s ban and telephone modules are good places to start). But, most Core tests set up their test fixtures in a completely different file than the test itself, which can be confusing. In this blog post, I’ll walk you through writing tests that look a bit more like the steps we’ve been doing manually. Automated migration tests don’t strictly require a Drupal 7 site at all, because the D7 testing tools in Core’s Migrate Drupal module know how to set up something that looks just enough like D7 to make the tests work. In order to run PHPUnit, you will need to set up the Drupal 9 site a bit differently than you may be accustomed to — the composer create-project commands (or the tar/zip files) you normally use when creating a Drupal site will not install the tools we need for running tests. We should clone Drupal core from source if we want to use PHPUnit. If we are going to write tests, we should seriously consider sharing them with the community, either by pushing the tests to an Issue fork, or by generating a patch and interdiff that includes them. While we won’t actually generate a patch this week, the instructions below will get you to set up your environment as if you were going to generate a patch. Clone Drupal core’s 9.2.x branch, set up your development environment on the repository (note there is no web/ or html/ folder in this setup), and run composer install. Find a contrib module that has a migration patch (as described in part 2 of this blog series). As before, read through the issue with the patch in detail. Clone the 8.x version of the module into your D9 site, as described in part 2. Apply the migrations patch to its own branch the 8.x module and commit the contents of the patch to the branch; or checkout the Issue fork, as described in part 2. If the issue is using patches, then before you continue, you should create a second branch to add your tests in — the changes in this second branch will become your patch; and running git diff between the first and second branches will become your interdiff. To do this: Check out the branch from the issue “Version” field again (e.g.: git checkout 8.x-2.x) Create a new branch to put your work in. I name this branch after the issue ID and the number of comments in the issue plus one. For example, if the issue number is 123456, and there are currently 8 comments in the issue (i.e.: the comment number of the most recent comment is #8), then I would name my branch 123456-9. Apply the patch again — but this time, don’t commit the changes yet (you need to add your tests first). Before we write a test, we need to take a closer look at the migration that we want to test. Recall from the migration patches that migrations are defined by YAML files inside the module’s migrations/ directory. These files have roughly the following structure… # In a file named migrations/MIGRATION_NAME.yml... id: MIGRATION_NAME label: # a human-friendly name migration tags: - Drupal 7 # possibly more tags source: plugin: # a @MigrateSource plugin id # some config for that @MigrateSource plugin process: # some process config destination: plugin: # a @MigrateDestination plugin id # some config for that @MigrateDestination plugin Right now, we only need to know the MIGRATION_NAME from the id line for one of the Drupal 7 migrations. If you find several migrations in the migrations/ folder, I’d suggest starting with a configuration migration, because those are usually the simplest. Create a folder for the tests: mkdir -p tests/src/Kernel/Migrate/d7 Using your preferred text editor, create a PHP file in that folder, tests/src/Kernel/Migrate/d7/MigrateTest.php, and edit it as follows, replacing MODULE_NAME with the machine name of the module; and MIGRATION_NAME with the migration name you found in the migrations/MIGRATION_NAME.yml file you’re going to test… namespace Drupal\Tests\MODULE_NAME\Kernel\Migrate\d7; use Drupal\Tests\migrate_drupal\Kernel\d7\MigrateDrupal7TestBase; /** * Test the MIGRATION_NAME migration. * * @group MODULE_NAME */ class MigrateTest extends MigrateDrupal7TestBase { /** * {@inheritdoc} */ protected static $modules = ['MODULE_NAME']; /** * Test the MIGRATION_NAME migration. */ public function testMigration() { // TODO: Set up fixtures in the source database. // Run the migration. $this-executeMigrations(['MIGRATION_NAME']); // TODO: Verify the fixtures data is now present in the destination site. // TODO: Remove this comment and the $this-assertTrue(TRUE); line after it once you've added at least one other assertion: $this-assertTrue(TRUE); } } Let’s run the test: php core/scripts/run-tests.sh --sqlite /tmp/test.sqlite --file modules/MODULE_NAME/tests/src/Kernel/Migrate/d7/MigrateTest.php This assumes php is in your shell’s $PATH, you’ve changed directories to the path containing Drupal 9’s index.php, you can write temporary files to /tmp/, and you installed the module you’re patching to modules/MODULE_NAME. If you’re using Lando or Ddev, you will probably need to lando ssh -s appserver or ddev ssh -s web before running the line above. If all goes well, you should see output like… Drupal test run --------------- Tests to be run: - Drupal\Tests\MODULE_NAME\Kernel\Migrate\d7\MigrateTest Test run started: Tuesday, August 24, 2021 - 13:00 Test summary ------------ Drupal\Tests\MODULE_NAME\Kernel\Migrate\d7\MigrateTest 1 passes Test run duration: 5 sec But the test isn’t very useful yet. Exactly how to fill in the TODOs we’ve left in there depends on the specific module you’re working on (i.e.: the data it stored in D7, and how that data maps to D9). For now, let’s look at a real-world example: migrating the configuration for the Environment Indicator module (note there’s already a migration to do that in issue #3198995 — please do not create a new issue, and please do not leave patches in that issue). To keep this blog post (relatively) short, I will provide a sample migration definition to migrate two pieces of configuration in environment_indicator. We will discuss how to find data to migrate and how to write migration definitions in future blog posts in this series. Looking at the code in the latest D7 release, I see 2 pieces of config to migrate: environment_indicator_integration, and environment_indicator_favicon_overlay. Suppose that someone has written following migration definition at migrations/d7_environment_indicator_settings.yml to migrate those 2 pieces of config: id: d7_environment_indicator_settings label: Environment indicator settings migration_tags: - Drupal 7 - Configuration source: plugin: variable variables: - environment_indicator_integration - environment_indicator_favicon_overlay source_module: environment_indicator process: toolbar_integration: environment_indicator_integration favicon: environment_indicator_favicon_overlay destination: plugin: config config_name: environment_indicator.settings You can see here that the MIGRATION_NAME in our template can be filled in with d7_environment_indicator_settings. So let’s start by copying the migration test template above into the file tests/src/Kernel/Migrate/d7/MigrateTest.php, and replacing MIGRATION_NAME with d7_environment_indicator_settings. Now, since these two pieces of config were stored in the variable table in D7; we will start by inserting those variables into the variable table through the migrate database connection (i.e.: the source database)… // TODO: Set up fixtures in the source database. \Drupal\Core\Database\Database::getConnection('default', 'migrate') -insert('variable') -fields([ 'name' = 'environment_indicator_integration', 'value' = serialize(['toolbar' = 'toolbar']), ]) -execute(); \Drupal\Core\Database\Database::getConnection('default', 'migrate') -insert('variable') -fields([ 'name' = 'environment_indicator_favicon_overlay', 'value' = serialize(TRUE), ]) -execute(); Looking at the D9 version of environment_indicator, I can see global config is stored in the environment_indicator.settings config object; and there are two global settings in that object — toolbar_integration and favicon — whose behaviour matches the D7 variables we found. So let’s test the config after the migration: // TODO: Verify the fixtures data is now present in the destination site. $this-assertSame(['toolbar' = 'toolbar'], $this-config('environment_indicator.settings')-get('toolbar_integration')); $this-assertSame(TRUE, $this-config('environment_indicator.settings')-get('favicon')); Now let’s run the migration test that we’ve been filling in… $ php core/scripts/run-tests.sh --sqlite /tmp/test.sqlite --file modules/environment_indicator/tests/src/Kernel/Migrate/d7/MigrateTest.php Drupal test run --------------- Tests to be run: - Drupal\Tests\environment_indicator\Kernel\Migrate\d7\MigrateTest Test run started: Tuesday, August 24, 2021 - 13:05 Test summary ------------ Drupal\Tests\environment_indicator\Kernel\Migrate\d7\Migrate 1 passes Test run duration: 5 sec … great! Let’s clean up a bit by deleting the dummy assertion at the end and its comment (since we’ve added other assertions); and removing the remaining TODOs (since they are done). We can also add a use statement for Drupal\Core\Database\Database and modify the ::getConnection() lines accordingly. Now the full test looks like: namespace Drupal\Tests\environment_indicator\Kernel\Migrate\d7; use Drupal\Core\Database\Database; use Drupal\Tests\migrate_drupal\Kernel\d7\MigrateDrupal7TestBase; /** * Test the d7_environment_indicator_settings migration. * * @group environment_indicator */ class MigrateTest extends MigrateDrupal7TestBase { /** * {@inheritdoc} */ protected static $modules = ['environment_indicator']; /** * Test the d7_environment_indicator_settings migration. */ public function testMigration() { // Set up fixtures in the source database. Database::getConnection('default', 'migrate') -insert('variable') -fields([ 'name' = 'environment_indicator_integration', 'value' = serialize(['toolbar' = 'toolbar']), ]) -execute(); Database::getConnection('default', 'migrate') -insert('variable') -fields([ 'name' = 'environment_indicator_favicon_overlay', 'value' = serialize(TRUE), ]) -execute(); // Run the migration. $this-executeMigrations(['d7_environment_indicator_settings']); // Verify the fixtures data is now present in the destination site. $this-assertSame(['toolbar' = 'toolbar'], $this-config('environment_indicator.settings')-get('toolbar_integration')); $this-assertSame(TRUE, $this-config('environment_indicator.settings')-get('favicon')); } } Congratulations, you’ve written your first automated Migration test! In the next blog post, we’ll talk about migrating simple configuration (i.e.: D7 variables to D9 config objects). In the meantime, you could try refactoring the tests/src/Kernel/Migrate/d7/MigrateTest.php test we built in this blog post. Some ideas: Try splitting the Database::getConnection(...)-...-execute() statements into a helper function, Try randomizing the fixtures data that you insert, Try making two test methods, one for environment_indicator_favicon_overlay, where you test both the TRUE and FALSE states; and one for environment_indicator_integration. If this is your first time writing automated tests, you might be interested in reading PHPUnit’s documentation on writing tests. PHPUnit’s assertions reference can also be pretty handy to refer to when writing tests. If you have a lot of time, some optional, longer reads are: Drupal’s MigrateDrupal7TestBase class; api.drupal.org’s “automated tests” topic; Drupal.org’s “PHPUnit in Drupal” docs landing page; and; Drupal.org’s “Migration tests” docs landing page. https://consensus.enterprises/blog/migrations-2-rtbc/ Tue, 17 Aug 2021 09:00:00 +0000 Easy commit credits with migrations, part 2: Can we get an RTBC? on Consensus Enterprises Blog published Tue, 17 Aug 2021 09:00:00 +0000 This is the second in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In this part, I’ll cover how to review and manually test patches. In part 3, I demonstrate how to write automated migration tests. In part 4, I discuss migrating simple configuration. In part 5, I explain how to declare a module’s migration status to the migrate wizard. In part 6, I show how to migrate data from a custom database table. Stay tuned for more in … This is the second in a series of blog posts on writing migrations for contrib modules: In part 1, we set up a simple core migration. In this part, I’ll cover how to review and manually test patches. In part 3, I demonstrate how to write automated migration tests. In part 4, I discuss migrating simple configuration. In part 5, I explain how to declare a module’s migration status to the migrate wizard. In part 6, I show how to migrate data from a custom database table. Stay tuned for more in this series! While migrating off Drupal 7 Core is very easy, there are still many contrib modules without any migrations. Any sites built using a low-code approach likely use a lot of contrib modules, and are likely blocked from migrating because of contrib. But — as of this writing — Drupal 7 still makes up 60% of all Drupal sites, and time is running out to migrate them! If we are to make Drupal the go-to technology for site builders, we need to remember that migrating contrib is part of the Site Builder experience too. If we make migrating easy, then fewer site builders will put off the upgrade or abandon Drupal. Plus, contributing to migrations gives us the opportunity to gain recognition in the Drupal community with contribution credits. As a maintainer of several modules, it really helps me when other members of the community review and test patches, and if they work, mark them with the issue status RTBC (“Reviewed and Tested by the Community”)! One of the easiest ways that you can contribute is by testing migration patches as thoroughly as you can, reviewing the code, and marking the issue as RTBC if everything checks out. Any time you’re engaging with the Drupal community, and especially in the issue queue, it is worth keeping a few things in mind: The Drupal code of conduct — “be considerate; be respectful; be collaborative; when we disagree, we consult others; when we are unsure, we ask for help; and; step down considerately”; The Drupal.org issue etiquette — “dos” and “dont’s” for making issues flow smoother; and; The strategic initiative that you are working towards — in this case, improving the migration experience for Site Builders. These steps assume that you have followed the Steps to complete in in part 1 of this blog series at least once. First, we will need to find a contrib module with a migration patch in “Needs review” status. Read the issue with the patch in detail. Doing so will provide you with some insight into the intended scope of the patch, and also what to test. I suggest taking notes. For your convenience, here is a link to a search for the word migration across all projects, filtered to issues in “Needs review” status (but since this blog series is about contrib migrations, you can ignore the results for the project “Drupal core”)1. For example, let’s suppose that I found issue #3024040 in this list. Next, install the latest 7.x release of the contrib module you chose into the Drupal 7 site you set up in part 1 of this blog series (it is reasonable to assume that Site Builders looking to migrate are running the latest release of a contrib module). If you haven’t worked much with Drupal 7, the recommended place to install contrib modules is inside sites/all/modules/. Using our example issue #3024040; because it is a patch for the Tablefield module, I would install tablefield-7.x-3.6 into my D7 test site (because that was the latest recommended D7 version of the module at time-of-writing). Then, git clone the contrib module you chose into the web/modules/ folder of the Drupal 9 site you set up in part 1 of this blog series. You need to clone the branch specified in the “Version” field of the issue with the patch. For your convenience, Drupal.org can generate a git clone command for you to copy-paste: go to the module’s project page, and look at the top for a “Version control” tab… click that tab, choose the “Branch to work from”, and click the “Show” button. Using our example issue #3024040, the “Version” field in that issue’s metadata shows 8.x-2.x-dev, i.e. the 8.x-2.x Git branch. If we then click the module name (“Tablefield”) in the issue’s breadcrumb bar, then its Version control tab, then choose 8.x-2.x, and click “Show”, Drupal.org gives us the command git clone --branch '8.x-2.x' https://git.drupalcode.org/project/tablefield.git Next, we want to apply the most-recent patch in the issue — but before we do that, we should create a branch to apply the patch on (creating a branch will make it easier to generate interdiff files if we need to submit our own patch). If the issue is using an Issue fork instead of patches, then click “Show commands”, follow the instructions to “Add & fetch this issue fork’s repository”, then follow the instruction to “Check out this branch”, and skip ahead to the next step — the rest of the instructions in this step are for issues using patches. If the issue is using patches, then I usually create a branch named after the issue ID and comment number that I got the patch from. In our example issue #3024040, the most recent patch at time-of-writing is in comment #8, so I would name the branch 3024040-8, i.e.: I would run git checkout -b 3024040-8 Now we can apply the patch. Important note: if you’re following along with the example in issue #3024040, be aware that at some point in the future, the maintainers of the Tablefield module will likely accept and commit the patch — trying to apply the patch after it has been committed will fail. If the patch applies successfully, commit the changes in the patch to the new branch. There’s no need to come up with a fancy commit message because we won’t be pushing it anywhere: I use branch name as the commit message (e.g.: git commit -m "3024040-8") Now, we run through essentially the same process we used in part 1 of this blog series to test the migration. That is to say: (Re-)install the D7 site using the Standard install profile. (Re-)install the D9 site using the Standard install profile. On the D7 site, install the Tablefield module, and set it up (i.e.: add a Tablefield to a node type). Then, create some Tablefield nodes as test migration content. On the D9 site, install the core Migrate, Migrate Drupal, and Migrate Drupal UI modules; and also install the Tablefield module. Make a database backup of the D9 site (so you can easily re-run the migration). On the D9 site, run through the migration wizard at /upgrade. When the upgrade is complete, check the migrated Tablefield nodes to ensure they contain the test migration content you set up on the D7 site. If the test migration content you set up on the D7 site did not correctly migrate onto the D9 site, see the “What to do if something goes wrong” section below. If the migration appeared to go correctly, then read the patch in more detail. Future blog posts in this series should make the patch easier to understand, but even now, you can probably get a vague sense of what is being migrated, and how it is being done. In particular, if you notice that the patch migrates some things that you did not test, it would be worth reverting to the database backup you made, and trying the migration again, so you can test those new things. If you find coding style issues in a contrib patch, I would refrain from pointing them out — let the module maintainer do that if they feel strongly enough about it! Many Contrib maintainers have their own style, or don’t feel strongly about coding style: the coding standards used for Drupal Core are only suggestions for Drupal Contrib. Furthermore, some module maintainers will accept the patch but fix the style issues in the patch when they commit it (this is what I do for modules that I maintain). Remember the strategic initiative we are working towards: we want to improve the experience for Site Builders — attaining coding standards perfection will delay the patch and prevent it from helping the Site Builders who need it! Finally, if you are satisfied with the patch after reading it and testing the migration, then it is time to add a comment to the issue: In the Issue metadata, set the “Status” field to Reviewed and tested by the community, and make sure the “Assigned” field is set to Unassigned. Don’t forget to “Attribute this contribution”. In the “Comment” field, clearly state that it worked for you, and describe what you tested. Finally, click “Save” to post your comment. Note that the patch to Tablefield in issue #3024040 is just used as an example — please do not leave comments in that issue unless you have something specific and constructive to add. If the test migration doesn’t go the way you expect, this may not necessarily indicate a problem with the patch! For example: The D9 version of the module may operate or store data in a different way than the D7 version does; The D9 version of the module may have fewer or different features from the D7 version; The issue that you got the patch from intentionally leaves certain migrations out-of-scope; or; Your expectations might be wrong (this happens to me a lot!). So, before leaving a comment in the issue, take some time to: Read the issue in depth (including earlier patch versions and interdiffs, if applicable), to understand what is, and is not, in scope; Skim the D7 and the D9 versions of the module’s code, to understand the differences between the two versions of the module and how they work; and; Read the patch, to understand what it is trying to migrate and to try to pinpoint the problem. If you think that you can pinpoint the problem, then it’s worth posting your own comment on the issue. In your comment: Describe the steps you took, Describe how to create the content and/or configuration which did not migrate properly in D7, Explain what you expected the migrated content and/or configuration to look like in D9, Explain what you think the problem is. Be aware that the patch author and/or module maintainer may be okay with things not working perfectly! Recall the Drupal code of conduct: be respectful (of the module maintainer’s decisions), and step down (i.e.: back off) considerately. As mentioned earlier, the best way to find Migration issues that need review is to search for them. As you may know, marking your own patches RTBC is discouraged, so you’ll probably run across patches that I’ve written floating out there in the issue queues! If you’re reading through an issue, and you find it confusing to keep track of everything that changed, other people probably find it confusing too! You can help move the issue forward by simply updating the issue summary. But, be aware that, like coding standards, following Core’s issue sumamry template is just a suggestion in Contrib. In the next blog post, we’ll talk about converting some of the testing that you’re doing manually right now. I’ve proposed officially adopting the migrate issue tag for contrib migration issues, but this needs to be approved by the Drupal.org administrators, so don’t tag issues with it for now. I’ll update this blog post if this proposal is accepted. ↩︎ https://consensus.enterprises/blog/migrations-1-migrate-core/ Tue, 10 Aug 2021 09:00:00 +0000 Easy commit credits with migrations, part 1: Migrating Drupal Core on Consensus Enterprises Blog published Tue, 10 Aug 2021 09:00:00 +0000 This is the first in a series of blog posts on writing migrations for contrib modules: In this part, we set up a simple core migration. In part 2, I cover how to review and manually test patches. In part 3, I demonstrate how to write automated migration tests. In part 4, I discuss migrating simple configuration. In part 5, I explain how to declare a module’s migration status to the migrate wizard. In part 6, I show how to migrate data from a custom database table. Stay tuned for more in this … This is the first in a series of blog posts on writing migrations for contrib modules: In this part, we set up a simple core migration. In part 2, I cover how to review and manually test patches. In part 3, I demonstrate how to write automated migration tests. In part 4, I discuss migrating simple configuration. In part 5, I explain how to declare a module’s migration status to the migrate wizard. In part 6, I show how to migrate data from a custom database table. Stay tuned for more in this series! Besides helping small teams do big things at Consensus Enterprises during the week, I also work part-time for a small family business. Naturally, I built the business’ website in Drupal (initially Drupal 6, now Drupal 7), and, like many other small sites, I took a low-code approach by assembling its functionality using configuration and 63 contrib projects. To make the best use of my budget, I focused my custom development efforts on making a unique theme. But, 8 years on, the website’s theme is looking a bit dated, Drupal 7 is quickly reaching the end of free support, and the small family business I built it for doesn’t have a budget for paid Drupal 7 support. I want to upgrade the site to Drupal 9. Happily, Drupal 8 and 9 have a simple Drupal-to-Drupal migration wizard, and the community has a fantastic Core migration team, so migrating from D7 Core to D8+ Core is easy and has a great user experience! (seriously — the past and present Core migration team deserves a lot of credit!) While migrating Drupal 6 or 7 Core to Drupal 8 or 9 Core is extremely easy, there are still many contrib modules without any migrations. Any sites built using a low-code approach likely use a lot of contrib modules — and are therefore blocked from migrating because of contrib! For my small business website, when I went through the migration wizard, and reached the “What will be upgraded?” step, I saw 104 “Modules that will not be upgraded”, compared with only 32 “Modules that will be upgraded”1! Honestly, I felt pretty discouraged when I saw that, even though I have the knowledge and experience to write the migrations myself! I don’t think it would be too much of a stretch to imagine that other Site Builders faced with the same situation might consider putting off the upgrade, or abandoning Drupal for a closed hosting platform. As of this writing, the official Usage statistics for Drupal Core show that there are around 600,000 other Drupal 7 sites — Drupal 7 still powers about 60% of all Drupal sites! (This fact should, perhaps, garner more attention than it does.) Anyway, During the April 2021 DrupalCon, when Dries said that we need to go back to our Site Builder roots and make Drupal the go-to technology for site builders experience, he hit the nail on the head… Migrating contrib is part of the Site Builder experience too! Hopefully, your thought is, “I work with Drupal too, and I want to help the Drupal community achieve its strategic initiatives — Can I help?” The answer is, “Yes!” It is actually pretty easy to write migrations! This makes them a great way for you and/or your employer to gain recognition in the Drupal community through contribution credits. Plus, you’ll get some valuable experience with both Drupal 7 and 8. In this blog series, I will walk you through the ways that I contribute to improve the migration experience for Site Builders, so that you can do those things too! Hopefully, by combining our efforts, you and I can make things easier for anyone who needs to migrate a Drupal 7 site to Drupal 8 or 9 (myself included), and help the Drupal community to achieve our strategic initiatives! This blog series will be geared towards people who: know how to download and install Drupal 8 or 9 and contrib modules on their workstation (with or without composer), know how to apply a patch, are comfortable doing a little bit of development work (mainly writing YAML files), are self-motivated to read publicly-available documentation and code, want to help, and, don’t mind engaging with the Drupal community. Let’s start off this blog series with something easy: setting up test sites and running a simple migration of Drupal core. First, you need to set up a Drupal 7 site to be your migration source. If you haven’t set up Drupal 7 before, it’s pretty easy: download the tarball or zip from the Core project page, extract it into a folder, set your HTTP Server (ideally Apache) to serve the folder containing index.php (there is no web/ or html/ folder in D7), and visit the site in a web browser to begin the installation process (which looks a lot like the Drupal 8 or 9 install process). Install the D7 site using the Standard install profile in the first step. You’ll also need to set up a Drupal 9 test site to be your migration destination. This blog series assumes you already know how to do this. Install the D9 site using the Standard install profile in the first step. Drupal 6 has much less usage than Drupal 7 (about 17,000 sites, or about 1.7% of Drupal’s total market share, as of this writing), and it requires a version of PHP between 4.3.5 and 5.3. You can certainly test migrations with Drupal 6 if you want, but be aware that it has already reached end-of-life and has known security vulnerabilities (so only install it where it can be accessed by people you trust, i.e.: not visible to the Internet), and while its install process is similar to D7’s, its PHP version requirements are not, and switching PHP versions will make it annoying to work with. Once you’ve got the Drupal 7 site set up, enable some modules (core modules for now - we’ll talk about contrib modules in a future post), configure them, and create a small amount of content. Keep track of what you’re doing, so you can see how it gets migrated later. Here’s where it gets interesting! On your Drupal 9 site, At /admin/modules, install core’s Migrate (migrate), Migrate Drupal (migrate_drupal), and Migrate Drupal UI (migrate_drupal_ui) modules (in the “Migration” group). Make sure any other modules that you installed/enabled on the Drupal 7 site (i.e.: when you were creating data to migrate) are also installed/enabled on the D9 site. If you wish, make a database backup of the D9 site, so that you can easily re-run the migration process (at time-of-writing, rollbacks were not yet supported through the user interface - although you could run them from the command-line). Start the migration wizard by going to /upgrade: Read the first page and click Continue On the second page, select Drupal 7 as the version of Drupal you’re migrating from, choose the database type (note MariaDB is roughly equivalent to MySQL), and enter the D7 site’s database connection information (i.e.: so the D9 site can connect to the D7 site’s database directly). Set up the Source files section if applicable, and click Review upgrade. If you’re using [Ddev][ddev], make sure both the D7 and D9 projects are running; then the “Database host” should be ddev-D7_PROJECT_NAME-db, and the “Port number” should be 3306 If you’re using Lando, make sure both the D7 and D9 apps are running; then the “Database host” should be database.D7_APP_NAME.internal, and the “Port number” should be 3306 If you see an “Upgrade analysis report”, read it and click I acknowledge... (multilingual migrations can be flakey) Read the “What will be upgraded?” report, and click Perform upgrade If you enabled some modules on D7, but you didn’t enable the corresponding modules in D9 before starting the upgrade process, they will appear as “Modules that will not be upgraded” here. Contrib modules without migrations — what this blog series is intended to help change — will appear as “Modules that will not be upgraded” in this step of the migration wizard. When the upgrade is complete, you’ll be returned to the site’s front page. Congratulations, you’re done: you can now explore the D9 site and see the users, content, configuration, etc. that was migrated from the D7 site! If you go back to /upgrade, you’ll see “An upgrade has already been performed on this site”, and a button to “Import new configuration and content from the old site”, i.e.: things that had changed on the D7 site since the last migration. If you’d like to test out some already-working contrib migrations, try out the Recipe module. In the next blog post, we’ll talk about reviewing migration patches. To close off this blog post, I’d like to propose that the Drupal.org issue tag maintainers add an official migration issue tag to the list of official issue tags. A migration tag would make it a lot easier for Site Builders to find patches for their modules; and for contributors to write and review those patches. You can weigh in on this proposal in issue #3227012 I thought you said “63 contrib projects” earlier! Where did 103 modules come from? Recall that a project can have sub-modules/sub-themes - turns out 25 were sub-modules. 17 more were Features (i.e.: what we used before the Configuration Management Initiative). Another 5 were custom modules - I’ve gotta write migrations for those ones myself. ↩︎ https://consensus.enterprises/blog/introducing-config-enforce/ Wed, 21 Apr 2021 11:40:00 -0400 Introducing Config Enforce on Consensus Enterprises Blog published Wed, 21 Apr 2021 11:40:00 -0400 Over the last few years we’ve built lots of Drupal 8 sites, and some Drupal 9 ones too, both for our clients and for ourselves. As such, we’ve taken a keen interest in (read: faced many challenges with) the Configuration Management subsystem. This was a major new component in Drupal 8, and so, while it’s functional, it isn’t yet mature. Of course, the vibrant Drupal developer community jumped in to smooth the rough edges and fill the gaps, in what has since become known as CMI 2.0. At … Over the last few years we’ve built lots of Drupal 8 sites, and some Drupal 9 ones too, both for our clients and for ourselves. As such, we’ve taken a keen interest in (read: faced many challenges with) the Configuration Management subsystem. This was a major new component in Drupal 8, and so, while it’s functional, it isn’t yet mature. Of course, the vibrant Drupal developer community jumped in to smooth the rough edges and fill the gaps, in what has since become known as CMI 2.0. At Consensus, we tend to work on fairly large, complex Drupal projects that often require significant custom development. As such, we’ve adopted fairly rigourous software engineering processes, such as standardized local development environments, CI-enabled Test Driven Development, Continuous Delivery, etc. However, we struggled to find a workflow that leveraged the powerful CMI system in core, while not being a pain for developers. The core CMI workflow assumes you are transferring configuration between a single site install with multiple instances (Dev, Stage, Prod) and that this configuration is periodically synchronized as a lump. For a number of reasons, this wouldn’t work for us. As a result, we went back to our old standby, the venerable Features module, which worked reasonably well. Unfortunately, we found that it would sometimes handle dependencies between configuration objects poorly. On more than one occasion, this led to time-consuming debugging cycles. So we switched to using Config Profile instead. However, reverting config changes was still manual, so we started using Config Update and the related Update Helper. The Update Helper module, “offers supporting functionalities to make configuration updates easier.” Basically, when preparing for a release, Update Helper generates a special file, a “configuration update definition” (CUD). The CUD contains two values for each changed config. The first is the “current” value for a given configuration, as of the most recent release. The second is the new value to which you want to set that config. These values are captured by first rolling back to the most recent release, then installing the site, so that the value is in active config. Then you checkout your latest commit, so that the new values are available on the filesystem. Update Helper can then generate its CUD, as well as generate a hook_update() implementations to help deploy the new or changed config. This process turned out to be error-prone, and difficult to automate reliably. We explored other efforts too, like Config Split and Config Filter which allow for finer-grained manipulation of “sets” of config. Other projects, like Config Distro, are focused on “packaging” groups of features such that they can be dropped in to any given site easily (kind of like Features…) A simple, reliable method to deploy new or updated configuration remained elusive. Note that all the tools mentioned above work very well during initial project development, prior to production release. However, once you need to deploy config changes to systems in production Update Helper or similar tools and processes are required, along with all the overheads that implies. At this point, it’s worth reminding ourselves that Drupal 7 and earlier versions did not clearly distinguish between content and config. They all just lived in the site’s database, after all. As such, whatever configuration was on the production site was generally considered canonical. It’s tempting to make small changes directly in production, since they don’t seem to warrant a full release, and all the configuration deployment overhead that entails. This, in turn, requires additional discipline to reproduce those changes in the codebase. Of course, that isn’t the only reason for configuration drift. Well-meaning administrators cannot easily distinguish between configs that are required for the proper operation of the site, and those that have more cosmetic effects. Facing these challenges, we’d regularly note how much easier all of this would be if only we could make production configuration read-only. With some reluctance and much consideration, we decided to try an entirely new approach. We built Config Enforce (and its close companion Config Enforce Devel) to solve the two key problems we were running into: Developers needed an easy way to get from “I made a bunch of config-related changes in the Admin UI of my local site instance” to “I can identify the relevant config objects/entities which have changed, get them into my git repository, and push them upstream for deployment”. Operations needed an easy way to deploy changes in configuration, and ideally not have to worry too much about the previously-inevitable “drift” in the production-environment configuration, which often resulted in tedious and painful merging of configs, or worse yet, inadvertent clobbering of changes. Config Enforce has two “modes” of operation: with config_enforce_devel enabled, you are in “development mode”. You can quickly designate config objects you want to enforce (usually inline on the configuration page where you manipulate the configuration object itself), and then changes you make are immediately written to the file system. This mode leverages Config Devel to effectively bypass the active configuration storage in the database, writing the config files into target extensions you select. Each target extension builds up a “registry” of enforced configuration objects, keeping track of their location and enforcement level. This eases the development workflow by making it easy to identify which configuration objects you’ve changed without having to explictly config-export and then identify all and only the relevant .yml files to commit. In production mode, you enable only the config_enforce module, which leverages the same “registry” configuration that config_enforce_devel has written into your target extensions, and performs the “enforcement” component. This means that, for any enforced configuration objects, we block any changes from being made via the UI or optionally even API calls directly. In turn, the enforced configuration settings on the file system within target extensions become authoritative, being pulled in to override whatever is in active configuration whenever a cache rebuild is triggered. This means that deployment of configuration changes becomes trivial: commit and push new enforced configuration files in your target extensions, pull those into the new (e.g., Prod) environment, and clear caches. configuration Enforce will check all enforced configuration settings for changes on the file system, and immediately load them into active configuration on the site. This workflow requires some adjustment in how we think about configuration management, but we think it has promise. Especially if you are building Drupal distributions or complex Drupal systems that require repeatable builds and comprehensive testing in CI, you should give Config Enforce a try and see what you think. Feedback is always welcome! We’ve scratched our own itch, and so far have found it useful and productive. We are pleased to make it available to the Drupal community as another in the arena of ideas surrounding CMI 2.0. https://consensus.enterprises/blog/goldilocks-bdd/ Sun, 21 Mar 2021 22:55:00 -0400 Goldilocks and the Three Projects on Consensus Enterprises Blog published Sun, 21 Mar 2021 22:55:00 -0400 How does Goldilocks know what she likes and doesn’t like? If you were making her a chair, a bed, or a bowl of porridge, how would you figure it out? How would you convince her to spend the time and resources needed to determine what is “juuust right” for her? We will use the traditional fairy tale as a jumping-off point for a discussion of testing first, failing fast, Behaviour-Driven Development and other agile practices we use at Consensus Enterprises. Along the way, we will review Behat, … How does Goldilocks know what she likes and doesn’t like? If you were making her a chair, a bed, or a bowl of porridge, how would you figure it out? How would you convince her to spend the time and resources needed to determine what is “juuust right” for her? We will use the traditional fairy tale as a jumping-off point for a discussion of testing first, failing fast, Behaviour-Driven Development and other agile practices we use at Consensus Enterprises. Along the way, we will review Behat, its Mink extension, and our own Drumkit framework: a BDD toolchain for Drupal that can help you make sure what you are building is what your customers actually want. A few of the topics you’ll learn about: The value of testing first and failing fast, focusing on Behaviour-Driven Development Drupal tools including Behat, Mink and Consensus Enterprises’ Drumkit How to make sure you’re developing using a customer-first approach The approach that Goldilocks takes is too haphazard. We don’t want agile development to look like a random walk through a strange environment. Things get broken that way! In this presentation, we introduce our approach to agile project requirements, using Behaviour Driven Development. This approach combines Business Analysis and Test Driven Development into a robust process that allows us to: Focus on what the customer wants and Ensure that we are providing it The BDD approach includes writing tests at the level of the observed behaviour of the software, so that we can confirm that what we are doing is what the customer expects. This allows the conversation with the customer to stay focused on the performance of the software, and not what is going on under the hood. We also include the (age-old) caution: “You can’t solve a people problem with a technical solution." Even with robust approaches to testing, the success of a project still depends on upfront investment and ongoing stakeholder engagment. One of the most important parts of a successful agile implementation is the presence of a Product Owner who is available throughout the process and empowered to make decisions on behalf of the client. For more details, I invite you to watch the recording of my presentation on YouTube. (Presentation starts at 15:46) My presentation slidedeck can be downloaded here: Goldilocks and the Three Projects! https://consensus.enterprises/blog/meet-gary/ Tue, 09 Mar 2021 13:00:00 -0400 Developing a Low-Cost High-Impact Site on Consensus Enterprises Blog published Tue, 09 Mar 2021 13:00:00 -0400 One of our core principles is solidarity with each other and with our communities. As a part of living this principle, Consensus takes on occasional projects that have the potential for high impact but lack the necessary budget for custom development. We have some standard templates for such projects, so we can spin them up quickly for review by the clients, and deploy them to low- or no-cost hosts, appropriate to the amount of traffic they are expecting. Our most recent project in this category … One of our core principles is solidarity with each other and with our communities. As a part of living this principle, Consensus takes on occasional projects that have the potential for high impact but lack the necessary budget for custom development. We have some standard templates for such projects, so we can spin them up quickly for review by the clients, and deploy them to low- or no-cost hosts, appropriate to the amount of traffic they are expecting. Our most recent project in this category was a website to help refugees to Canada navigate the Refugee Board process. The MeetGary website was created by a team of refugee lawyers and refugee law researchers to consolidate and promote their research about how the preconceptions of members of the refugee board can put claimants at risk of being incorrectly rejected. They emphasized communicating their research in clear language and using stories. They created storylines for over 50 typical claimants, with a wide range of backgrounds and situations. The cases provide examples of the most common misunderstandings and mistakes, and insights into how to avoid or counteract them. Additionally, the MeetGary team has provided two peer-reviewed journal articles that can be included as evidence in the refugees’ claims. Because this is an information site, with only occasional updates and no need to capture user information, we decided to build it using a Hugo static site generator. We use this approach for our own internal documentation and the documents embedded in several of our software projects, so we’re familiar with it as a lightweight means of producing websites quickly. (They also happen to be very fast, but that’s not an issue on a site of this size.) This approach lets us use a CI pipeline to make and test all changes locally and push the completed website to GitLab pages, which provides free hosting for static sites. Using this standardized approach across all our projects (even the small ones) makes our development work simpler, but it also means that if a build fails, it will never get promoted to the live site. Additionally, Hugo provides native support for multi-language sites through the simple use of a consistent file structure and naming convention. Each page is translated separately, but the entire navigation system (including internal links) is self-updating. Adding new languages is a relatively straightforward process of putting the files in the right place with a two-letter code in the filename. So this-file-en.md (for example) points to the English translation and this-file-es.md would appear in the Spanish (Español) menu. Reused pieces of text can also be added to the i18n configuration files. This allows the use of shortcodes for things like navigation arrows. In this case, we used the feature to turn the Go back [to another page] and Go forward [to page] to learn more into complete sentences in all the supported languages, allowing the same functionality (and friendliness) for all users. It turns out that the dev tools I am familiar with are not great at parsing bidirectional text. Not only that, there are outstanding requests for this support that are years old. It’s not something that I’ve ever had to think about before, but (as an aside) I am annoyed on behalf of my colleagues for whom this is an ongoing issue that this has gone unfixed for this length of time. This incorporation of RTL languages (Persian and Arabic) was the main challenge with this project, since none of our developers speak or read the languages in question. As a result, getting this right required checking in with translators in mid-project. Even though the various tools could manage bidirectional text as long as it occurred in the middle of sentences, we found that embedded shortcodes or HTML tags at the beginning of paragraphs could cause the whole paragraph to cut-and-paste incorrectly (and inconsistently). Also, the text editor (the one I’m using now, as it happens) mirrored all the RTL text in display, which was then turned back around by the addition of the languagedirection: rtl parameter in the hugo configuration file. I tried three text editors before giving up and implementing these changes manually, checking rendered blocks of text against the translated documents and sending screen shots of my local implementation back to the translators. If I had it to do over again, I would spend some time finding a native RTL editor for those pages, because this particular issue added about 20 unexpected hours to the project. In the end, we and the client are very pleased with the end result, which provides a much-needed service to a highly vulnerable group. Please share widely to maximize its benefit in the world! For more details about this project and its associated organizations, check out the “About Us” page on the project site: https://meetgary.ca/about-us/ https://consensus.enterprises/blog/kubernetes-wont-save-you/ Wed, 27 Jan 2021 08:06:00 +0000 Kubernetes Won't Save You on Consensus Enterprises Blog published Wed, 27 Jan 2021 08:06:00 +0000 A lot of potential clients come to us with straightforward and small projects and ask, “Well, can you do Kubernetes?” And we say, “Well, we can, but you don’t need it.” But they’re afraid that they’ll be missing out on something if we don’t add Kubernetes to the stack. So this is a post to tell you why we probably won’t be recommending Kubernetes. This post is going to look at three perspectives on this question… First, I’ll consider the technical aspects, specifically what problems Kubernetes … A lot of potential clients come to us with straightforward and small projects and ask, “Well, can you do Kubernetes?” And we say, “Well, we can, but you don’t need it.” But they’re afraid that they’ll be missing out on something if we don’t add Kubernetes to the stack. So this is a post to tell you why we probably won’t be recommending Kubernetes. This post is going to look at three perspectives on this question… First, I’ll consider the technical aspects, specifically what problems Kubernetes is really good for, compared with what problems most smaller software projects actually have. Then I’ll talk about the psychology of “shiny problems.” (Yes, I’m looking at you, Ace Developer. I promise there are other shiny problems in the project you’re working on.) And last but not least, we’ll consider the business problem of over-engineering, and what gets lost in the process. First off, (I’m sorry to draw your attention to this, but): You probably don’t have the problems that Kubernetes solves. Kubernetes lives at the container orchestration level. It shines in its ability to spin up and down stateless servers as needed for load balancing unpredictable or pulsed loads, especially from a large user base. Large, by the way, is not 10,000… it is millions or hundreds of millions. Especially for the kind of internal custom services that a lot of our clients require, it is overkill. Many purpose-built sites are unlikely to have more than dozens or hundreds of users at a time, and traditional monolithic architectures will be responsive enough. Kubernetes is designed to solve the problem of horizontal scalability, by making multiple copies of whichever services are most stressed, routing requests to minimize latency, and then being able to turn those machines back off when they are no longer needed. Even if you hope to someday have those problems, we suggest that you should hold off on adding Kubernetes to your stack until you get there, because the added technical overhead of container orchestration is expensive, in both time and dollars. (It costs more to build, which delays your time to market, which delays your time to revenue, even if you aren’t paying yourself to build it.) Which does lead to the question, “Why does everybody want to use this technology, anyway?” For that, we’ll have to take a step back and look at… With the shift to the cloud and the desire for highly scalable applications, a new software architecture has arisen that has a strong separation between a system’s code and its data. This approach treats processes as stateless and independent, and externalizes the database as a separate “backing service.” The stateless processes are isolated as microservices, which are each maintained, tested, and deployed as separate code bases. This microservices approach decomposes the software into a group of related but separate apps, each of which is responsible for one particular part of the application. Designing according to this architectural approach is non-trivial, and the overhead associated with maintaining the separate code bases, and particularly in coordinating among them is significant. Additionally, each app requires its own separate datastore, and maintaining synchronization in production introduces another level of complexity. Furthermore, extracting relevant queries from distributed systems of data is more challenging than simply writing a well-crafted SQL statement. Each of these layers of complexity adds to the cost of not only the initial development, but also the difficulty of maintenance. Even Chris Richardson, in Microservices Patterns, recommends starting with a monolithic architecture for new software to allow rapid iteration in the early stages. (https://livebook.manning.com/book/microservices-patterns/chapter-1/174) For many of the same reasons, you probably don’t need complex layers of data handling either. Redis, for example, is for persisting rapidly changing data in a quickly accessible form. It’s not suitable for a long-standing database with well-established relations, it costs more to run data in memory than to store it on disk, and it’s more difficult to build. When you are getting started, a SQL back end with a single codebase will probably solve most of your problems, and without the overhead of Kubernetes (or any of the more exotic data stores.) If you’re still not convinced, let’s take a brief detour and consider the lifecycle of a typical application. Most applications have the following characteristics: Predictable load Few (fewer than millions of) users Predictable hours of use (open hours of the business, daily batch processing cron job at 3 AM, etc.) Clear options for maintenance windows Tight connection between the content layer and the presentation layer Contrast this with the primary assumptions in the twelve-factor app approach. The goal of moving to stateless servers is focused on different things: Zero downtime Rapid development Scalability This approach arose from the needs of large consumer-facing applications like Flickr, Twitter, Netflix, and Instagram. These need to be always-on for hundreds of millions or billions of users, and have no option for things like maintenance mode. When we apply the Dev-Ops calculus to smaller projects, though, there is an emphasis on Dev that comes at the expense of Ops. Even though we may include continuous integration, automated testing and continuous deployment (and we strongly recommend these be included!), the design and implementation of the codebase and dependency management often focuses on “getting new developers up and running” with a simple “bundle install” (or “build install” etc.) This is explicitly stated as a goal in the twelve-factor list. This brings in several tradeoffs and issues in the long-term stability of the system; in particular, the focus on rapid development comes at a cost for operations and upgrades. The goal is to ship things and get them standing up from a cold start quickly… which is the easy part. The more difficult part of operations – the part you probably can’t escape, because you probably aren’t Netflix or Flickr or Instagram – is the maintenance of long-standing systems with live data. Upgrades of conventional implementations proceed thusly: Copy everything to a staging server Perform the upgrade on the staging server If everything works, port it over to the production environment There are time delays in this process: for large sites it can take hours to replicate a production database to staging, and if you want a safe upgrade, you need to put the site into maintenance mode to prevent the databases from diverging. The staging environment, no matter how carefully you set it up, is rarely an exact mirror of production; the connections to external services, passwords, and private keys for example, should not be shared. Generally, after the testing is complete in the staging environment, the same sequence of scripts is deployed in production. Even after extensive testing, it may prove necessary to roll back the production environment, database and all. Without the use of a maintenance freeze, this can result in data loss. This sort of upgrade between versions is significantly easier in monolithic environments. It’s tempting to point to Kubernetes’ rolling updates and the ability to connect multiple microservices to different pods of the database running the different versions… but in content-focused environments, the trade-off for zero downtime is an additional layer of complexity required to protect against potential data loss. Kubernetes and other 12-factor systems resolve the issue of data protection by sharding and mirroring the data across multiple stores. The database is separate from the application, and upgrades and rollbacks proceed separately. This is a strength for continuous delivery, but it comes at a cost: data that is produced in a blue environment during a blue-green deployment may simply be lost if it proves necessary to roll back schema changes. Additionally, if there are breaking changes to the schema and the microservices wind up attached to a non-backward compatible version, they can throw errors to the end-user (this is probably preferable to data loss.) For data persistence, the data needs to be stored in volumes externally from the K8 cluster, and orchestrating multiple versions of the code base and database simultaneously requires significant knowledge and organization. A deployment plan for such a system will need to include plans for having multiple versions of the code live on different servers at the same time, each of which connects to its associated database until the upgrade is complete and determined to be stable. It can be done, but even Kubernetes experts point out that this process is challenging to oversee. When we are moving things into production, we need to have an operations team that knows how to respond when something fails. No matter how much testing you have done, sometimes a Big Hairy Bug gets into production, and you need to have enough control of your system to be able to fix it. Kubernetes, sad to say, makes this harder instead of easier for stateful applications. So let’s consider what it means to have a stateful application. A content management system by its nature is stateful. A stateful application has a lot of data that makes up a large fraction of “what it is.” State can also include cache information, which is volatile, but the data is part and parcel of what we are doing. Databases and the application layer are frequently tightly integrated, and it’s not meaningful to ship a new build without simultaneously applying schema updates. The data itself is the point of the application. Drupal (for example) contains both content and configuration in the database, but there is additional information contained in the file structure. These, in combination, make up the state of the system… and the application is essentially meaningless without it. Also, as in most enterprise-focused applications, this data is not flat but is highly structured. The relationships are defined by both the database schema and the application code. It is not the kind of system that lends itself to scaling through the use of stateless containers. In other words: by their very nature, Drupal applications lack the strict separation between database and code that makes Kubernetes an appropriate solution. One of the things that (we) engineers fall into is a desire to solve interesting problems. Kubernetes, as one of the newest and most current technologies, is the “Shiny” technology towards which our minds bend. But it is complex, has a steep learning curve, and is not the first choice when deploying stateful applications. This means that a lot of the problems you’re going to have to solve are going to be related to the containers and Kubernetes/deployment layer of the application, which will reduce the amount of time and energy you have to solve the problems at the data model and the application layer. We’ve never built a piece of software that didn’t have some interesting challenges; we promise they are available where you are working. Also, those problems are probably what your company’s revenues rely on, so you should solve them first. As I hope I’ve convinced you, the use of heavier technologies than you need burns through your resources and has the potential to jeopardize your project. The desire to architect for the application you hope to have (rather than the one you do) can get your business into trouble. You will need more specialized developers, more complex deployment plans, additional architectural meetings and more coordination among the components. When you choose technologies that are overpowered (in case you need them at some undefined point in the future), you front-load your costs and increase the risk that you won’t make it to revenue/profitability. We get it. We love good tech as much as the next person. The fact is, though, most projects don’t need response times measured in the millisecond range. They just need to be fast enough to keep users from wandering away from the keyboard while their query loads. (Or they need a reasonable queuing system, batch processing, and notification options.) And even if you do need millisecond response times but you don’t have millions of users, Kubernetes will still introduce more problems than it solves. Performance challenges like these are tough, but generally need to be solved by painstaking, time-consuming, unpredictable trial and error–and the more subcomponents your application is distributed/sharded into, the harder (more time consuming, more unpredictable - by orders of magnitude!) that trial and error gets. Most sites are relatively small and relatively stable and will do quite well on a properly-sized VM with a well-maintained code base and a standard SQL server. Minimizing your technological requirements to those that are necessary to solve the problems at hand allows you to focus on your business priorities, leaving the complexity associated with containerization and the maintenance of external stateful information to a future iteration. Leave the “How are we going to scale” problem for once you get there, and you increase the chances that this will eventually be the problem you have. https://consensus.enterprises/blog/protecting-cloud-networks-wireguard-ansible/ Fri, 24 Jul 2020 11:02:26 -0400 Protecting your cloud networks with WireGuard VPN and Ansible on Consensus Enterprises Blog published Fri, 24 Jul 2020 11:02:26 -0400 Within cloud computing, there are various types of sites and services not meant for public consumption (e.g. analytics software, databases, log servers, etc.). For security reasons, it’s best to keep these accesssible only via the private network, which is behind the firewall. To provide access to these resources, a virtual private network (VPN) should be used, with network access granted only to trusted individuals within the organization. Traditionally, OpenVPN, IPsec and other solutions were … Within cloud computing, there are various types of sites and services not meant for public consumption (e.g. analytics software, databases, log servers, etc.). For security reasons, it’s best to keep these accesssible only via the private network, which is behind the firewall. To provide access to these resources, a virtual private network (VPN) should be used, with network access granted only to trusted individuals within the organization. Traditionally, OpenVPN, IPsec and other solutions were the go-to options within the open-source software space. However, these are often complex to set up and have relatively massive code bases making them difficult to maintain. For example, OpenVPN requires that a certificate authority (CA) be set-up. This is a complex piece of software, which shouldn’t be necessary for running a VPN. WireGuard simply requires the exchange of public keys in order to set up a secure connection, much like SSH and PGP. There’s been a lot of interest in WireGuard lately, notably because: It was recently added to the Linux kernel. NordVPN, one of the major VPN service providers, has started using it. Mozilla, the company behind the Firefox Web browser, just started offering it as a service. It’s recommended in The Definitive 2019 Guide to Cryptographic Key Sizes and Algorithm Recommendations However, in the Why Not WireGuard article, some opposition was raised. Let’s tackle some of the points raised there. Fine, but we’re doing the opposite. The clients can have dynamic IP addresses, but the server never will. So it’s irrelevant for this use case. That’s precisely the purpose of this article: To introduce an easy way to set it up and maintain it with Ansible. Lacking cipher agility is actually a good thing. A better approach is to use versioned protocols. And it’s actually no more difficult to upgrade WireGuard clients than anything else. Both of these non-issues are discussed very nicely in the article Against Cipher Agility in Cryptography Protocols. As it was recently added to the Linux kernel, this is no longer an issue. While performance can always be improved, this doesn’t appear to be a critical issue for the application. For most use cases, it’s perfectly usable. None of the above “issues” are actually a problem here. Ansible allows for automated deployment of configuration, which removes the need for manually installing, configuring and maintaining applications. It provides tonnes of modules, including those for files, storage, system, networking and even cloud provisioning (although I would generally recommend Terraform for this purpose). Its units of work are called “tasks” that are run sequentially (procedural) in “roles” and “playbooks” to perform operations such as installing server software, its configuration, and handling various other types of system administration. Ansible strives for simplicity, resulting in playbooks that are essentially self-documenting. It can safely be run multiple times (as it strives to be idempotent), running tasks only when necessary, leaving already-configured items as-is. While there were several WireGuard roles available for installing and maintaining the application, they either: didn’t cater to the cloud gateway VPN use case, lacked documentation, and/or intentionally omitted critical elements (e.g. packet forwarding to internal hosts) for implementation by the user. As such, I’ve written a comprehensive one. It’s packaged as a collection as this is the newer distribution format that doesn’t concern itself with the location of source control repositories. Traditionally, it was necessary for roles to be hosted on GitHub for them to be published on Galaxy, the site for sharing Ansible contributions. As I prefer GitLab for hosting code repositories, this seemed more natural. The project is therefore hosted on GitLab.com. The collection contains the single WireGuard role, and can be installed with ansible-galaxy (Ansible 2.9+). For older versions of Ansible, simply clone the Git repository and create a symbolic link to roles/wireguard_cloud_gateway within it. Documentation can be found in the role’s README. Some of the other roles I researched didn’t provide much support for configuring the client side of the VPN, meaning the devices which connect to the cloud gateway server to access private network resources. My role, on the other hand, can be run in either client or server mode: the same role can be used for configuring both. Running it in server mode configures the server (on the gateway VM), and running it in client mode configures the client devices (who connect to the server to gain access to the private network). For cloud security akin to traditional firewalls, security groups are essential for protecting virtual-machine (VM) compute instances. While these can be configured manually, ideally such configuration would be infrastructure as code (IaC) implemented via a tool such as Terraform, stored in a version control system (VCS) such as Git. In a typical VPN-server set-up, the incoming (“ingress”) rules for such a “VPN” security group would block access to all ports except the one upon which the VPN communicates. This configuration should be applied to the VM that will be running WireGuard. However, if this is the case, using Ansible to install it won’t work because Ansible uses SSH to connect to the VM, and the SSH port is blocked. In order to allow for such a secure set-up, a security group ID (e.g. “public_ssh”) can be provided to the role as a variable, which will be used to temporarily allow SSH access. Once the installation is complete, this temporary access will be revoked. For those of us that rely on VPN technology, it’s often necessary to connect to multiple VPNs at the same time, or at least prevent network resource IDs from overlapping. For example, you want to avoid having two VMs on different networks from having the same IP address. WireGuard supports this by allowing multiple interfaces. By default, wg0 is used as the first one, but wg1, wg2, etc. can all coexist. If each remote network can exist on a different subnet, there’s no conflict from the client perspective. For example: wg0 can be used to access network A, with subnet 10.1.0.0/16 wg1 can be used to access network B, with subnet 10.2.0.0/16 To set this up in your Ansible playbook when calling the role, set the service_interface variable. By default, it’s wg0. Also, be sure to set the client_accessible_ips properly; this defines the subnet. For details, see the default variables file. The role was originally written for specific systems, notably OpenStack networks and Ubuntu VMs. However, we’d like to see the role support as many systems as possible (e.g. the IaaS platforms Amazon Web Services (AWS), Microsoft’s Azure, Google Cloud Platform (GCP), Digital Ocean, etc. and non-Debian-based operating systems (OSes) such as CentOS, Red Hat, etc.). If you work with these systems, we’re more than happy to accept your supporting code via merge requests. Otherwise, if you’re able to provide funding, we can add support for these systems on your behalf. To get in touch with us for that, or for any other reason, please use our contact form. We provide consulting in several areas, such as: Enterprise cloud architecture Cloud computing infrastructures as a service (IaaS) Infrastructure automation with Terraform Automating full-stack configuration with Ansible OpenStack, Amazon Web Services (AWS), Google Cloud Platform (GCP) & Microsoft Azure consulting Virtual private networking (VPNs) with OpenVPN and WireGuard https://consensus.enterprises/blog/drupal-hosting-company-lacking-composer-support/ Thu, 12 Mar 2020 15:33:00 -0400 Does your Drupal hosting company lack native Composer support? on Consensus Enterprises Blog published Thu, 12 Mar 2020 15:33:00 -0400 Best practices for building Web sites in the Drupal framework (for major versions 8 and above) dictate that codebases should be built with the Composer package manager for PHP. That is, the code repository for any sites relying on it should not contain any upstream code; it should only contain a makefile with instructions for assembing it. However, there are some prominent Drupal hosting companies that don’t support Composer natively. That is, after receiving updates to Composer-controlled Git … Best practices for building Web sites in the Drupal framework (for major versions 8 and above) dictate that codebases should be built with the Composer package manager for PHP. That is, the code repository for any sites relying on it should not contain any upstream code; it should only contain a makefile with instructions for assembing it. However, there are some prominent Drupal hosting companies that don’t support Composer natively. That is, after receiving updates to Composer-controlled Git repositories, they don’t automatically rebuild the codebase, which should result in changes to the deployed code. If you’re hosting your site(s) at one of these companies, and you have this problem, why not consider the obvious alternative? Aegir, the one-and-only open-source hosting system for Drupal that’s been around for over 10 years, has had native Composer support for over 2 years. That is, on each and every platform deployment (“platform” is Aegir-speak for a Drupal codebase), Aegir reassembles the upstream code assets by running the following automatically: composer create-project --no-dev --no-interaction --no-progress As a result, any sites created on that platform (or migrated/upgraded to it) will have access to all of the assets built by Composer. Additionally, Aegir now ships with the Aegir Deploy module, which enhances the platform creation process. It allows for the following types of deployment: Classic/None/Manual/Unmanaged Drush Makefile deployment Pure Git Composer deployment from a Git repository Composer deployment from a Packagist repository For more information, please read the Deployment Strategies section of the documentation. If you’d like to get started with Aegir, the best option would be to spin up an Aegir Development VM, which allows you to run it easily, play with it, and get familiar with the concepts. Naturally, reading the documentation helps with this too. Afterwards, review the installation guide for more permanent options, and take advantage of our Ansible roles. We have a policy role that configures the main role using our favoured approach. For help, contact the community, or get in touch with us directly. We provide the following Aegir services: Installation & maintenance in corporate/enterprise (or other) environments Architectural and technical support Hosting guidance Coaching Audits Upgrades Conversion to best practices https://consensus.enterprises/blog/aegir5-is-coming/ Tue, 10 Mar 2020 13:45:00 -0400 Aegir 5 is coming, and not just for Drupal! on Consensus Enterprises Blog published Tue, 10 Mar 2020 13:45:00 -0400 Aegir is the one-and-only FLOSS hosting system for Drupal sites that’s been around for over 10 years, a rock in the community. While Drupal hosting companies have come and gone, Aegir’s always been there for folks who want to host Drupal sites themselves. According to recent data at the time of this writing, there are 567 instances (that we know about). It’s used by organizations worldwide such as the US National Democratic Institute, NASA, and the European Commission. While Aegir 3 is the … Aegir is the one-and-only FLOSS hosting system for Drupal sites that’s been around for over 10 years, a rock in the community. While Drupal hosting companies have come and gone, Aegir’s always been there for folks who want to host Drupal sites themselves. According to recent data at the time of this writing, there are 567 instances (that we know about). It’s used by organizations worldwide such as the US National Democratic Institute, NASA, and the European Commission. While Aegir 3 is the currently stable recommended major release, we’ve started working on Aegir 5, which is a complete rewrite. It has notable differences such as: Drush, traditionally used as the provisioner, has been replaced by Ansible, which allows for the hosting of any type of site or service, not just Drupal. The front-end, formerly Drupal 7, has been replaced by Drupal 8, which allows us to take advantage of all of the newer features it provides. Components are now best-of-breed open-source tools such as Celery, for the task queue. When Aegir was originally written, tools such as Ansible and Celery didn’t exist so all of the functionality was written as Aegir-specific code. We can now get off that island. The entire project is now maintained in a single code repository, unlike the traditional four (Hosting, Provision, Hostmaster and Eldir) that have been maintained historically. An out-of-the-box framework for automatic site updates via the Distributions concept. This was experimental in Aegir 3. While still maintaining Aegir 3, we’d like to direct any new development initiatives towards the more modern Aegir 5. The initial focus is on supporting Drupal. We then intend to add support for Matomo, Hugo, and other applications we use. However, documentation has been started on how to add support for anything else so merge requests or funding for new apps are greatly encouraged. Along with the main documentation site, which includes the architecture, there is also an issues board for tracking tickets. While we’re working on it as quickly as we can, we sometimes get delayed by other priorities. As such, we’re actively looking for sponsors to help us prioritize development. Please get in touch if you’re interested in partnering, collaborating, providing funding, or anything else. https://consensus.enterprises/blog/install-openstack-cli-ubuntu-1910/ Mon, 09 Mar 2020 18:32:00 -0500 Installing the OpenStack CLI on Ubuntu 19.10 (Eoan), 20.04 (Focal) or later on Consensus Enterprises Blog published Mon, 09 Mar 2020 18:32:00 -0500 When working with OpenStack as an infrastructure-as-a-service (IaaS) cloud-computing platform, it’s rather convenient to be able to interface with it via its command-line interface (CLI). While the service is typically installed on the Ubuntu Long-Term Support (LTS) operating system (OS), which has releases every two years, running the CLI from other OSes, such as interim Ubuntu releases, is often necessary. However, it is currently not possible to install the command-line client with supported … When working with OpenStack as an infrastructure-as-a-service (IaaS) cloud-computing platform, it’s rather convenient to be able to interface with it via its command-line interface (CLI). While the service is typically installed on the Ubuntu Long-Term Support (LTS) operating system (OS), which has releases every two years, running the CLI from other OSes, such as interim Ubuntu releases, is often necessary. However, it is currently not possible to install the command-line client with supported Debian packages on Ubuntu 19.10. Update (2020-05-30): The simplest way to install the OpenStack CLI nowadays, at least on Ubuntu 20.04 (Focal) and later, is via the Snap package. Debian packages are no longer made available: sudo snap install openstackclients --classic This should be all you need to do; there’s no reason to follow the instructions in the remaining portion of this article. Update (2021-08-18): You may run into the following error on Ubuntu 21.04: % openstack versions show Traceback (most recent call last): File "/usr/local/bin/openstack", line 5, in from openstackclient.shell import main ModuleNotFoundError: No module named 'openstackclient' If you do, run the following commands: sudo apt install pip sudo pip install python_openstackclient Openstack commands should start working again. Attempting to follow the instructions in the documentation for the two listed OpenStack releases, which require an additional software archive, results in the message: cloud-archive for Rocky only supported on bionic cloud-archive for Stein only supported on bionic In addition, attempting to use the native software repository is impossible because the package isn’t available. As such, I’ve written a short script to fetch and install all of the necessary Debian packages from another release. #!/bin/sh DEST_DIR=/tmp/openstack-cli-debs mkdir $DEST_DIR wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-netaddr/python-netaddr_0.7.19-1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-iso8601/python-iso8601_0.1.11-1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/pyinotify/python-pyinotify_0.9.6-1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-monotonic/python-monotonic_1.5-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.context/python-oslo.context_2.22.1-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-warlock/python-warlock_1.2.0-2_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/s/stevedore/python-stevedore_1.30.1-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-os-service-types/python-os-service-types_1.6.0-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.config/python-oslo.config_6.8.1-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-debtcollector/python-debtcollector_1.20.0-2_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.log/python-oslo.log_3.42.3-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-os-client-config/python-os-client-config_1.31.2-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-requestsexceptions/python-requestsexceptions_1.4.0-1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-deprecation/python-deprecation_2.0.6-1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-dogpile.cache/python-dogpile.cache_0.6.2-6_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-json-patch/python-jsonpatch_1.21-1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-munch/python-munch_2.3.2-1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-openstackclient/python-openstackclient_3.18.0-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-cinderclient/python-cinderclient_4.1.0-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-cliff/python-cliff_2.14.1-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-glanceclient/python-glanceclient_2.16.0-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-keystoneauth1/python-keystoneauth1_3.13.1-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-keystoneclient/python-keystoneclient_3.19.0-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-neutronclient/python-neutronclient_6.11.0-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-novaclient/python-novaclient_13.0.0-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-openstacksdk/python-openstacksdk_0.26.0-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-osc-lib/python-osc-lib_1.12.1-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.i18n/python-oslo.i18n_3.23.1-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.serialization/python-oslo.serialization_2.28.2-0ubuntu1_all.deb wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.utils/python-oslo.utils_3.40.3-0ubuntu1_all.deb sudo dpkg -i $DEST_DIR/* sudo apt --fix-broken install Hopefully this helps other folks who have also run into this issue. https://consensus.enterprises/blog/add-hugo-docs-to-gitlab-project/ Thu, 13 Feb 2020 09:00:00 -0400 How to add a Hugo-based Docs site to your GitLab Project on Consensus Enterprises Blog published Thu, 13 Feb 2020 09:00:00 -0400 At Consensus, we spend a fair bit of our time building complex Drupal systems and the infrastructure that supports them. In that context, the simplicity of a tool like Hugo caught my attention. Hugo is a static site generator written in Go that has proven elegant and flexible for many situations where a simple (or even slightly complex) website is called for, and the venerable Drupal is overkill. Perhaps one of the best uses we’ve found for this tool is to provide technical documentation for a … At Consensus, we spend a fair bit of our time building complex Drupal systems and the infrastructure that supports them. In that context, the simplicity of a tool like Hugo caught my attention. Hugo is a static site generator written in Go that has proven elegant and flexible for many situations where a simple (or even slightly complex) website is called for, and the venerable Drupal is overkill. Perhaps one of the best uses we’ve found for this tool is to provide technical documentation for a development project. Hugo reduces the website to a handful of plain text files (some Markdown, some YAML, some HTML/CSS), so it’s easy to drop it into a sub-folder of the main project, meaning it lives alongside the code of the site/application itself. This makes it easier to maintain and refer to in the course of regular development workflow. Here’s a quick rundown of how to get started incorporating a Hugo docs site into your existing development project. Hugo Setup Create some content Serve it up (locally) GitLab Pages Conclusion Once you Install Hugo on your computer, creating a new site is easy: initialize a new site, grab a theme, and do some basic configuration. The Learn theme is a reasonable starting point for a docs site. Assuming you are starting from an existing Git project at $PROJECT_ROOT, do: cd $PROJECT_ROOT hugo new site docs git submodule add https://github.com/matcornic/hugo-theme-learn.git docs/themes/learn git add docs git commit -m "Initialize docs site" cd docs # Your docs site lives in here TOML is the default format, but we prefer YAML. We will replace the default config.toml with a config.yaml. The template below is a good starting point, but note that you’ll need to replace the GROUP, and PROJECT parameters to match your GitLab project. You can also fill in various details like Title, Description, Author, and various feature flags. Note: If your project is in a GitLab sub-group, the baseURL and editURL parameters below will need to include the slug for the sub-group path. ie. http://GROUP.gitlab.io/SUBGROUP/PROJECT. config.yml: baseUrl: "http://GROUP.gitlab.io/PROJECT/" languageCode: "en-US" defaultContentLanguage: "en" title: "My Project Docs Site" theme: "learn" metaDataFormat: "yaml" defaultContentLanguageInSubdir: true params: editURL: "https://gitlab.com/GROUP/PROJECT/tree/master/docs/content/" description: "Description of project docs site" author: "Consensus Enterprises" showVisitedLinks: true disableBreadcrumb: false disableNextPrev: false disableSearch: false disableAssetsBusting: false disableInlineCopyToClipBoard: false disableShortcutsTitle: false disableLanguageSwitchingButton: false ordersectionsby: "weight" # or "title" menu: shortcuts: - name: " Gitlab repo" url: "https://gitlab.com/GROUP/PROJECT" weight: 10 - name: " Contributors" url: "https://gitlab.com/GROUP/PROJECT/graphs/master" weight: 30 # For search functionality outputs: home: - "HTML" - "RSS" - "JSON" Now, remove the original config.toml and add the new config.yaml to git: git rm config.toml git add config.yaml git commit -m "Configure learn theme for docs site" One extra thing we’ll set up is to enable site-wide search of the docs. This is accomplished by adding a JSON index of the entire site, described in the layouts folder. Create the file docs/layouts/index.json: [{{ range $index, $page := .Site.Pages }} {{- if ne $page.Type "json" -}} {{- if and $index (gt $index 0) -}},{{- end }} { "uri": "{{ $page.Permalink }}", "title": "{{ htmlEscape $page.Title}}", "tags": [{{ range $tindex, $tag := $page.Params.tags }}{{ if $tindex }}, {{ end }}"{{ $tag| htmlEscape }}"{{ end }}], "description": "{{ htmlEscape .Description}}", "content": {{$page.Plain | jsonify}} } {{- end -}} {{- end -}}] Once again, add this new file to git: git add layouts/index.json git commit -m "Add index.json layout for site-wide search index" Now you can start creating some content. Again, these are just simple text files in Markdown format, so you can create and edit them using whatever your favourite text editor may be. The content/ folder within your Hugo site contains all content posts, and the menu structure will mirror whatever folder hierarchy you develop. To get started quickly, you can use hugo to initialize some posts for you: hugo new _index.md to create the front page hugo new technical/_index.md to create a sub-section hugo new technical/architecture.md to create a sub page This will set up the file for the post, which includes metadata (what Hugo calls the “front matter”), after which you simply edit it, and add the content. The front matter of a Hugo post is a section at the top of the file containing metadata such as title, date, etc. By default, Hugo will create new posts in “draft” state, so at minimum you want to edit each new file to “publish” it. Edit the files you created above (content/_index.md and content/technical/architecture.md), adjust the title as needed, set draft: false, and add some content. For example: --- title: "Technical Architecture" date: 2019-11-06T16:58:13-05:00 draft: false --- This page describes the technical architecture of the project. Make sure your new content files are added to git: git add content git commit -m "Add some content" Now you are ready to see the site in action. One of the great things about locally is it’s trivial to spin up the site on your local machine and see your draft updates live: hugo serve point your browser to http://localhost:1313/PROJECT By default, Hugo in this mode will auto-refresh your browser page any time you save a change to a content file, so you can actually see your revisions taking effect on an instance of the site as you type. This makes it much easier to manage updating documentation for a project in tandem with building out or making changes to the codebase itself. As mentioned above, the menu structure for your docs site mirrors the folder hierarchy within the content/ directory. For example, here’s a typical “tree” of content for a docs site: docs/content ├── _index.md ├── admin │ ├── _index.md │ ├── updating.md │ └── workflows │ ├── _index.md │ ├── taxonomy.md ├── dev │ ├── architecture │ │ ├── _index.md │ │ ├── search_system.md │ │ └── static_content.md │ ├── _index.md │ └── setup │ ├── dev-workflow.md │ ├── _index.md │ └── theme-dev.md └── testing ├── demo.md ├── _index.md └── writing_tests.md This would result in a top-level menu like this: Home (/) Admin (/admin/) Development (/dev) Testing (/testing) The menu titles themselves are determined by the frontmatter in the _index.md files at each level. Typically this matches the title: element, but sometimes you need to override it, in which case you can specify the menuTitle explicitly: --- title: Development Workflows using GitLab menuTitle: Dev Workflows weight: 30 --- Our final step is to publish the site using GitLab Pages. This is simply a matter of adding a .gitlab-ci.yml at the root of your project with a pages stage that runs hugo to generate static HTML from your new docs/ folder. Essentially, you tell GitLab CI to use a Hugo container to render the docs/ folder into a set of static HTML/CSS/JS in the public subfolder, and then move that folder into a location from which GitLab Pages can serve it up. Note: Before you push to CI, it’s important that you set at least the baseURL in your docs/config.yaml file, so it matches where GitLab Pages will place it. See GitLab’s docs on Pages default domain names for details. Here’s a simple example .gitlab-ci.yml: stages: - publish variables: GIT_SUBMODULE_STRATEGY: recursive pages: stage: publish image: jojomi/hugo cache: {} script: - hugo -s docs # Render the Hugo site - mv docs/public . # Move the public folder into place artifacts: paths: - public # Tell GitLab Pages to serve the public folder only: - master See the CI/CD for GitLab Pages docs for more information on this, or the full .gitlab-ci.yml reference for all the gory details. Now you can push your changes up to GitLab, and you should immediately see a CI pipeline kick in to publish your Pages site. git add .gitlab-ci.yml git commit -m "Add .gitlab-ci.yml config to publish Pages site" git push Go to the project’s Settings Pages, and you will see a URL where the site is published under the *.gitlab.io domain, and you can also attach your own domain by clicking the New Domain link and following the instructions to point a DNS entry to the GitLab Pages server. A co-worker of mine has created an example repository by following the steps in this post and committing regularly. You can find it at gitlab.com/mparker17/learn-hugo. Documentation for a technical project is a crucial component for even the most simple cases. Hugo provides an excellent framework to provide documentation housed within the project’s codebase. It’s easy for developers to edit and update in tandem with their regular development workflow, and review those changes locally before pushing them up alongside code updates. GitLab Pages provides an easy mechanism for publishing the resulting site, and GitLab CI will auto-publish updates with every push. Check out the Hugo Documentation pages to find out more about how to use Hugo, manage content, tweak the theme templates, and more. https://consensus.enterprises/blog/drupal-saas-building-software-as-a-service-on-drupal/ Fri, 15 Nov 2019 15:35:00 -0500 Drupal North 2019: Drupal SaaS: Building software as a service on Drupal on Consensus Enterprises Blog published Fri, 15 Nov 2019 15:35:00 -0500 On Friday, June 14th, I presented this session at Drupal North 2019. That’s the annual gathering of the Drupal community in Ontario and Quebec, in Canada. As I realized I hadn’t yet posted this information yet, I’m doing so now. Session information: Are you (considering) building a SaaS product on Drupal or running a Drupal hosting company? Have you done it already? Come share your experiences and learn from others. Among other things, we’ll be discussing: Project vs. product business … On Friday, June 14th, I presented this session at Drupal North 2019. That’s the annual gathering of the Drupal community in Ontario and Quebec, in Canada. As I realized I hadn’t yet posted this information yet, I’m doing so now. Session information: Are you (considering) building a SaaS product on Drupal or running a Drupal hosting company? Have you done it already? Come share your experiences and learn from others. Among other things, we’ll be discussing: Project vs. product business Installation profiles / distributions Customer service (e.g. GitLab’s Service Desk) Hosting architecture (Drupal hosting companies vs. Aegir) Infrastructure (IaaS hosting providers: OpenStack vs. AWS, GCS, Azure, etc.) E-commerce, recurring billing and subscription provider integration Aegir Site Subscriptions Others? Resource quotas Site admin access permissions for clients …and any other related topics that come up. A video recording of my presentation is available on: Drupal.tv YouTube My slides (with clickable links) are available on our presentations site. https://consensus.enterprises/blog/lando-drumkit-setup/ Thu, 07 Nov 2019 23:06:00 +0000 Lando and Drumkit for Drupal 8 Localdev on Consensus Enterprises Blog published Thu, 07 Nov 2019 23:06:00 +0000 Over the last 2 or 3 years, the Drupal community has been converging around a solid set of Docker-based workflows to manage local development environments, and there are a number of worthy tools that make life easier. My personal favourite is Lando, not only because of the Star Wars geekery, but also because it makes easy things easy and hard things possible (a lot like Drupal). I appreciate that a “standard” Lando config file is only a few lines long, but that it’s relatively easy to configure … Over the last 2 or 3 years, the Drupal community has been converging around a solid set of Docker-based workflows to manage local development environments, and there are a number of worthy tools that make life easier. My personal favourite is Lando, not only because of the Star Wars geekery, but also because it makes easy things easy and hard things possible (a lot like Drupal). I appreciate that a “standard” Lando config file is only a few lines long, but that it’s relatively easy to configure and customize a much more complex setup by simply adding the appropriate lines to the config. In this post I want to focus on an additional tool I’ve come to lean on heavily that complements Lando quite nicely, and that ultimately boils down to good ol’ fashioned Makefiles. Last summer at DrupalNorth I gave a talk that was primarily about the benefits of Lando, and I only mentioned Drumkit in passing. Here I want to illustrate in more detail how and why this collection of Makefile tools is a valuable addition to my localdev toolbox. The key benefits provided by adding a Drumkit environment are: consistent make -based workflow to tie various dev tasks together ease onboarding of new devs (make help) make multistep tasks easier (make tests) make tasks in Lando or CI environment the same (ie. make install && make tests) This example is using Drumkit for a Drupal 8 localdev environment, but there’s no reason you couldn’t use it for other purposes (and in fact, we at Consensus have lately been doing just that. As an example, suppose you’re setting up a new D8 project from scratch. Following this slide from my Lando talk, you would do the basic Lando D8 project steps: Create codebase with Composer (composer create-project drupal-composer/drupal-project:8.x-dev code --stability dev --no-interaction) Initialize Git repository (git init etc.) Initialize Lando (lando init) For now, leave out the lando start step, which we’ll let Drumkit handle momentarily. We should also customize the .lando.yml a little with custom database credentials, which we’ll tell Drumkit about later. Append the following to your .lando.yml: services: database: creds: user: chewie_dbuser password: chewie_dbpass database: chewie_db To insert Drumkit into this setup, we add it as a git submodule to our project using the helper install.sh script, and bootstrap Drumkit: wget -O - https://gitlab.com/consensus.enterprises/drumkit/raw/master/scripts/install.sh | /bin/bash . d # Use 'source d' if you're not using Bash The install script checks that you are in the root of a git repository, and pulls in Drumkit as a submodule, then initializes a top-level Makefile for you. Finally, we initialize the Drumkit environment, by sourcing the d script (itself a symlink to .mk/drumkit) into our shell. Note that Drumkit will modify your PATH and BIN_PATH variables to add the project-specific .mk/.local/bin directory, which is where Drumkit installs any tools you request (eg. with make selenium. This means if you have multiple Drumkit-enabled projects on the go, you’re best to work on them in separate shell instances, to keep these environment variables distinct. Note that you can take advantage of this environment-specific setup to customize the bootstrap script to (for example) inject project credentials for external services into the shell environment. Typically we would achieve this by creating a scripts/bootstrap.sh that in turn calls the main .mk/drumkit, and re-point the d symlink there. Because we’re using Composer to manage our codebase, we also add a COMPOSER_CACHE_DIR environment variable, using the standard .env file, which Drumkit’s stock bootstrap script will pull into your environment: echo "COMPOSER_CACHE_DIR=tmp/composer-cache/" .env . d # Bootstrap Drumkit again to have this take effect From here, we can start customizing for Drupal-specific dev with Lando. First, we make a place in our repo for some Makefile snippets to be included: mkdir -p scripts/makefiles echo "include scripts/makefiles/*.mk" Makefile Now we can start creating make targets for our project (click the links below to see the file contents in an example Chewie project. For modularity, we create a series of “snippet” makefiles to provide the targets mentioned above: scripts/makefiles/variables.mk sets up some project variables scripts/makefiles/lando.mk provides targets to start, stop, poweroff, and destroy the Lando containers scripts/makefiles/build.mk provides targets to build and update the codebase scripts/makefiles/install.mk provides targets to install the Drupal site NB You’ll need to customize the variables.mk file with the DB credentials you set above in your .lando.yml as well as your site name, admin user/password, install profile, etc. Now our initial workflow to setup the project looks like this: git clone --recursive cd . d # or "source d" if you're not using Bash make start make build make install This will get a new developer up and running quickly, and can be customized to add whatever project-specific steps are needed along the way. But wait- it gets even better! If I want to make things really easy on fellow developers (or even just myself), I can consolidate common steps into a single target within the top-level Makefile. For example, append the make all target to your Makefile: .PHONY: all all: @$(MAKE-QUIET) start @$(MAKE-QUIET) build @$(MAKE-QUIET) install Now, the above workflow for a developer getting bootstrapped into the project simplifies down to this: git clone --recursive cd . d make all At this point, you can start adding your own project-specific targets to make common workflow tasks easier. For example, on a recent migration project I was working on, we had a custom Features module (ingredients) that needed to be enabled, and a corresponding migration module (ingredients_migrate) that needed to be enabled before migrations could run. I created the following make targets to facilitate that workflow: scripts/makefiles/migrate.mk provides make setup and make migrate targets We often take this further, adding a make tests target to setup and run our test suite, for example. This in turn allows us to automate the build/install/test process within our CI environment, which can call exactly the same make targets as we do locally. Ultimately, Drumkit is a very simple idea: superimpose a modular Makefile-driven system on top of Lando to provide some syntactic sugar that eases developer workflow, makes consistent targets that CI can use, and consolidates multi-step tasks into a single command. There’s lots more that Drumkit can do, and plenty of ideas we have yet to implement, so if you like this idea, feel free to jump in and contribute! https://consensus.enterprises/blog/drupalcamp-ottawa-2019-automate-all-the-things/ Thu, 24 Oct 2019 22:55:00 -0400 DrupalCamp Ottawa 2019: Automate All the Things on Consensus Enterprises Blog published Thu, 24 Oct 2019 22:55:00 -0400 On Friday, October 18th, I presented at DrupalCamp Ottawa 2019. That’s the annual gathering of the Drupal community in Ottawa, Ontario, Canada. Session information: Ever heard of infrastructure-as-code? The idea is basically to use tools like Ansible or Terraform to manage the composition and operation of your cloud systems. This allows infrastructure to be treated just like any other software system. The code can be committed into Git which allows auditability, and reproducibility. It can … On Friday, October 18th, I presented at DrupalCamp Ottawa 2019. That’s the annual gathering of the Drupal community in Ottawa, Ontario, Canada. Session information: Ever heard of infrastructure-as-code? The idea is basically to use tools like Ansible or Terraform to manage the composition and operation of your cloud systems. This allows infrastructure to be treated just like any other software system. The code can be committed into Git which allows auditability, and reproducibility. It can therefore be tested and integrated into full continuous delivery processes. Ansible provides tonnes of cloud management modules, from simple Linodes or Digital Ocean Droplets through globe-spanning AWS networks. Ansible also strives for simplicity, resulting in playbooks that are essentially self-documenting. In this session, we will: explore the principles of infrastructure-as-code and how to operationalize them; introduce Ansible and it’s cloud modules; build a full OpenStack cloud infrastructure end-to-end from scratch. A video recording of my presentation is available on YouTube. My presentation slidedeck can be downloaded here: Automate All the Things! https://consensus.enterprises/blog/exposing-drupal-taxonomy-data-sematic-web/ Thu, 24 Oct 2019 22:22:00 -0400 Exposing Drupal's Taxonomy Data on the Semantic Web on Consensus Enterprises Blog published Thu, 24 Oct 2019 22:22:00 -0400 As a content management framework, Drupal provides strong support for its taxonomical subsystem for classifying data. It would be great if such data could be exposed via the Simple Knowledge Organization System (SKOS) standard for publishing vocabularies as linked data. As Drupal becomes used more and more as a back-end data store (due to features such as built-in support for JSON:API), presenting this data in standard ways becomes especially important. So is this actually possible now? If not, … As a content management framework, Drupal provides strong support for its taxonomical subsystem for classifying data. It would be great if such data could be exposed via the Simple Knowledge Organization System (SKOS) standard for publishing vocabularies as linked data. As Drupal becomes used more and more as a back-end data store (due to features such as built-in support for JSON:API), presenting this data in standard ways becomes especially important. So is this actually possible now? If not, what remains to be done? First, let’s explore some of Drupal core’s history as it relates to the Semantic Web and Web services formats, also useful for future reference. This is basically the backstory that makes all of this possible. This was implemented in the (now closed) issues: Add a REST export display plugin and serializer integration Convert the taxonomy listing and feed at /taxonomy/term/%term to Views Here’s the change notice: Use schema.org types and properties in RDF mappings And a follow-up issue requesting support for additional namespaces: Allow usage of any namespace in RDF mapping UI Here’s an article with the details: Decoupling Drupal 8 Core: Web Services in Core and the Serialization Module As this is really a two-part issue, adding machine-readable metadata and then making machine-readable data available, I’ll split the discussion into two sections. While there’s an RDF UI module that enables one to specify mappings between Drupal entities and their fields with RDF types and properties, it only supports Schema.org via RDFa (not JSON-LD). As explained very well in Create SEO Juice From JSON LD Structured Data in Drupal, a better solution is to use the framework provided by the Metatag module (used by modules such as AGLS Metadata). The article introduces the Schema.org Metatag module, which uses the Metatag UI to allow users to map Drupal data to Schema.org, and exposes it via JSON-LD. So one solution would be to: Clone Schema.org Metatag, calling the new module SKOS Metatag. Replace all of the Schema.org specifics with SKOS. Rejoice. But after taking some time to process all of the above information, I believe we should be able to use the knowledge of the vocabulary hierarchy to add the SKOS metadata. We probably don’t need any admin UI at all for configuring mappings. Assuming that’s true, we can instead create a SKOS module that doesn’t depend on Metatag, but Metatag may still be useful given that it already supports Views. Exposing the site’s data can be done best though Views. I wouldn’t recommend doing this any other way, e.g. accessing nodes (Drupal-speak for records) directly, or through any default taxonomy links for listing all of a vocabulary’s terms. (These actually are Views, but their default set-ups are missing configuration.) A good recipe for getting this up & running, for both the list and individual items, is available at Your First RESTful View in Drupal 8. To actually access the data from elsewhere, you need to be aware of the recent API change To access REST export views, one now MUST specify a ?_format=… query string, which explains why some consumers broke. The JSON-LD format is, however, not supported in Core by default. There is some code in a couple of sandboxes, which may or may not work, that will need to be ported to the official module, brought up-to-date, and have a release (ideally stable) cut. See the issue JSON-LD REST Services: Port to Drupal 8 for details. Now, the Metatag solution I proposed in the previous section may work with Views natively, already exposing data as JSON-LD. If that’s the case, this JSON-LD port may not be necessary, but this remains to be seen. Also, accessing the records directly (without Views) may work as well, but this also remains to be seen after that solution is developed. Clearly, there’s more work to be done. While the ultimate goal hasn’t been achieved yet, at least we have a couple of paths forward. That’s as far as I got with pure research. Due to priorities shifting on the client project, I didn’t get a chance to learn more by reviewing the code and testing it to see what does and doesn’t work, which would be the next logical step. If you’ve got a project that could make use of any of this, please reach out. We’d love to help move this technology further along and get it implemented. What is the relationship between RDF, RDFa, Microformats and Microdata What is the difference between Microdata, RDFa & JSON-LD? Structured Data (JSON+LD Rich Snippets) JSON LD Schema API WissKI Ontology (OWL) DCAT RDF entity Drupal2RDF (This is brand new, but at the time of this writing, there’s not enough info/code yet to figure out what it’s trying to do.) There’s a module called Smart Glossary which allows you to have multilingual SKOS Thesauri on your Drupal site, but I don’t think it’s useful at all as it’s part of a suite of modules maintained by PoolParty, where they expect you to use their data store: Smart Glossary PoolParty Taxonomy Manager for Drupal Semantic Connector PowerTagging Options for SKOS integration in Drupal 8 How can I work with SKOS data? https://consensus.enterprises/blog/drupal8-hook-update-tricks/ Tue, 08 Oct 2019 11:00:00 +0000 Drupal 8 hook_update() Tricks on Consensus Enterprises Blog published Tue, 08 Oct 2019 11:00:00 +0000 In Drupal 7, hook_update()/hook_install() were well-established mechanisms for manipulating the database when installing a new site or updating an existing one. Most of these routines ended up directly running SQL against the database, where all kinds of state, configuration, and content data lived. This worked reasonably well if you were careful and had a good knowledge of how the database schema fit together, but things tended to get complicated. With the maturing of Features module, we were … In Drupal 7, hook_update()/hook_install() were well-established mechanisms for manipulating the database when installing a new site or updating an existing one. Most of these routines ended up directly running SQL against the database, where all kinds of state, configuration, and content data lived. This worked reasonably well if you were careful and had a good knowledge of how the database schema fit together, but things tended to get complicated. With the maturing of Features module, we were able to move some of this into configuration settings via the ctools-style export files, making the drush feature-revert command part of standard workflow for deploying new features and updates to an existing site. In Drupal 8, we’ve made huge strides in the direction of Object Orientation, and started to separate Configuration/State, Content Structure, and Content itself. The config/install directory is often all that’s needed in terms of setting up a contributed or custom module to work out of the box, and with the D8 version of Features, the same is often true of updates that involve straightforward updates to configuration .yml files. It turns out that both hook_update() and hook_install() are still valuable tools in our box, however, so I decided to compile some of the more complicated D8 scenarios I’ve run across recently. The hook_update_N API docs reveal that this function operates more or less as before, with some excellent guidelines for how to approach the body of the function’s implementation. The Introduction to update API handbook page provides some more detail and offers some more guidance around the kinds of updates to handle, naming conventions, and adding unit tests to the your update routines. The sub-pages of that Handbook section have some excellent examples covering the basics: Updating Configuration gives 3 examples of increasing complexity, demonstrating straightforward updates to configuration Updating Database Schema and/or Data covers changes to the structure of the database tables, and the data within them Updating Entities and Fields shows how to update entity- and field-level structure, making good use of the now-in-core Entity API (particularly useful with the recent removal of automated entity update magic) All of these provided a valuable basis on which to write my own real-life update hooks, but I found I still had to combine various pieces and search through code to properly write these myself. We recently launched our first complex platform based on Drupal 8 and the excellent OpenSocial, albeit heavily modified to suit the particular requirements of the project. The sub-profile required more extensive customization than simply extending the parent profile’s functionality (as discussed here). Instead, we needed to integrate new functionality into that provided by the upstream distribution, and this often resulted in tricky interactions between the two. Particularly with a complex site with many moving parts, we take the approach of treating the site as a system or platform, installing and reinstalling regularly via a custom installation profile and set of feature modules. This allows us to integrate: a CI system to build the system repeatedly, proving that everything works a Behat test suite to validate the behaviour of the platform matches the requirements In the context of a sub-profile of OpenSocial, this became complicated when the configuration we wanted to customize actually lived in feature modules from the upstream profile, and there was no easy way to just override them in our own modules’ config/install directories. We developed a technique of overriding entire feature modules within our own codebase, effectively forking the upstream versions, so that we could then modify the installed configuration and other functionality (in Block Plugins, for example). The trouble with this approach is that you have to manage the divergence upstream, incorporating new improvements and fixes manually (and with care). Thus, in cases where there were only a handful of configuration items to correct, we began using hook_install() routines to adjust the upstream-installed config later in the install process, to end up with the setup we were after. We make use of entity_legal for Terms of Service, Privacy Policy, and User Guidelines documents. Our installation profile’s feature modules create the 3 entity legal types, but we needed to be able to tweak the order of the form elements on the user/register page, which is a core entity_form_display created for the user entity. To achieve this using YAML files in the config/install directory per usual seemed tricky or impossible, so I wrote some code to run near the end of the installation process, after the new legal_entity types were created and the core user.register form display was set. This code simply loads up the configuration in question, makes some alterations to it, and then re-saves: /** * Implements hook_install(). */ function example_install() { _example_install_adjust_legal_doc_weights(); } /** * Adjust weights of legal docs in user/register form. */ function example_update_8001() { _example_install_adjust_legal_doc_weights(); } /** * Ensure the field weights on the user register form put legal docs at the bottom */ function _example_install_adjust_legal_doc_weights() { $config = \Drupal::getContainer()-get('config.factory')-getEditable('core.entity_form_display.user.user.register'); $content = $config-get('content'); $content['private_messages']['weight'] = 0; $content['account']['weight'] = 1; $content['google_analytics']['weight'] = 2; $content['path']['weight'] = 3; $content['legal_terms_of_service']['weight'] = 4; $content['legal_privacy_policy']['weight'] = 5; $content['legal_user_guidelines']['weight'] = 6; $config-set('content', $content)-save(); } A slightly more complicated situation is to alter a views configuration that is managed by an upstream feature module during the installation process. This is not an ideal solution, but currently it’s quite challenging to properly “override” configuration that’s managed by a “parent” installation profile within your own custom sub-profile (although Config Actions appears to be a promising solution to this). As such, this was the best solution I could come up with: essentially, run some code very nearly at the end of the installation process (an installation profile task after all the contrib and feature modules and related configuration are installed), that again loads up the views configuration, changes the key items needed, and then re-saves it. In this case, we wanted to add a custom text header to a number of views, as well as switch the pager type from the default “mini” type to “full”. This required some thorough digging into the Views API and related code, to determine how to adjust the “handlers” programmatically. This helper function lives in the example.profile code itself, and is called via a new installation task wrapper function, which passes in the view IDs that need to be altered. Here again, we can write trivial hook_update() implementations that call this same wrapper function to update existing site instances. /** * Helper to update views config to add header and set pager. */ function _settlement_install_activity_view_header($view_id) { # First grab the view and handler types $view = Views::getView($view_id); $types = $view-getHandlerTypes(); # Get the header handlers, and add our new one $headers = $view-getHandlers('header', 'default'); $custom_header = array( 'id' = 'area_text_custom', 'table' = 'views', 'field' = 'area_text_custom', 'relationship' = 'none', 'group_type' = 'group', 'admin_label' = '', 'empty' = '1', 'content' = '', 'plugin_id' = 'text_custom', 'weight' = -1, ); array_unshift($headers, $custom_header); # Add the list of headers back in the right order. $view-displayHandlers-get('default')-setOption($types['header']['plural'], $headers); # Set the pager type to 'full' $pager = $view-getDisplay()-getOption('pager'); $pager['type'] = 'full'; $view-display_handler-setOption('pager', $pager); $view-save(); } Of particular note here is the ordering of the Header components on the views. There was an existing Header on most of the views, and the new “Latest Activity” one needed to appear above the existing one. Initially I had tried creating the new custom element and calling ViewExecutable::setHandler method instead of the more complicated $view-displayHandlers-get('default')-setOption() construction, which would work, but consistently added the components in the wrong order. I finally found that I had to pull out a full array of handlers using getHandlers(), then array_unshift() the new component onto the front of the array, then put the whole array back in the configuration, to set the order correctly. In most cases we’ve been able to use Simple Block module to provide “custom” blocks as configuration, rather than the core “custom” block types, which are treated as content. However, in one case we inherited a custom block type that had relevant fields like an image and call-to-action links and text. Here again, the upstream OpenSocial modules create and install the block configs, and we didn’t want to fork/override the entire module just to make a small adjustment to the images and text/links. I came up with the following code block to effectively alter the block later in the installation process: First, the helper function (called from the hook_install() of a late-stage feature module in our sub-profile), sets up the basic data elements needed, in order to make it easy to adjust the details later (and re-call this helper in a hook_update(), for example): function _example_update_an_homepage_block() { ## CUSTOM ANON HOMEPAGE HERO BLOCK ## ## Edit $data array elements to update in future ## $data = array(); $data['filename'] = 'bkgd-banner--front.png'; # Lives in the images/ folder of example module $data['textblock'] = '' Sign up now to learn, share, connect and collaborate with leaders and those in related fields. '; $data['cta1'] = array( 'url' = '/user/register', 'text' = 'Get Started', ); $data['cta2'] = array( 'url' = '/about', 'text' = 'More about the Community', ); ## DO NOT EDIT BELOW THIS LINE! ## ################################## The rest of the function does the heavy lifting: # This code cobbled together from `social_core.install` and # `social_demo/src/DemoSystem.php` // This uuid can be used like this since it's defined // in the code as well (@see social_core.install). $block = \Drupal::entityTypeManager()-getStorage('block_content')-loadByProperties(['uuid' = '8bb9d4bb-f182-4afc-b138-8a4b802824e4']); $block = current($block); if ($block instanceof \Drupal\block_content\Entity\BlockContent) { # Setup the image file $fid = _example_setup_an_homepage_image($data['filename']); $block-field_text_block = [ 'value' = $data['textblock'], 'format' = 'full_html', ]; // Insert image file in the hero image field. $block_image = [ 'target_id' = $fid, 'alt' = "Anonymous front page image homepage'", ]; $block-field_hero_image = $block_image; // Set the links. $action_links = [ [ 'uri' = 'internal:' . $data['cta1']['url'], 'title' = $data['cta1']['text'], ], [ 'uri' = 'internal:' . $data['cta2']['url'], 'title' = $data['cta2']['text'], ], ]; $itemList = new \Drupal\Core\Field\FieldItemList($block-field_call_to_action_link-getFieldDefinition()); $itemList-setValue($action_links); $block-field_call_to_action_link = $itemList; $block-save(); } } The image helper function prepares the image field: function _example_setup_an_homepage_image($filename) { // TODO: use a better image from the theme. // Block image. $path = drupal_get_path('module', 'example'); $image_path = $path . DIRECTORY_SEPARATOR . 'images' . DIRECTORY_SEPARATOR . $filename; $uri = file_unmanaged_copy($image_path, 'public://'.$filename, FILE_EXISTS_REPLACE); $media = \Drupal\file\Entity\File::create([ 'langcode' = 'en', 'uid' = 1, 'status' = 1, 'uri' = $uri, ]); $media-save(); $fid = $media-id(); // Apply image cropping. $data = [ 'x' = 600, 'y' = 245, 'width' = 1200, 'height' = 490, ]; $crop_type = \Drupal::entityTypeManager() -getStorage('crop_type') -load('hero_an'); if (!empty($crop_type) && $crop_type instanceof CropType) { $image_widget_crop_manager = \Drupal::service('image_widget_crop.manager'); $image_widget_crop_manager-applyCrop($data, [ 'file-uri' = $uri, 'file-id' = $fid, ], $crop_type); } return $fid; } As with most things I’ve encountered with Drupal 8 so far, the Update system is both familiar and new in certain respects. Hopefully these concrete examples are instructive to understand how to adapt older techniques to the new way of managing install and update tasks. https://consensus.enterprises/blog/aegir-devops-deployment-workflows-drupal-sites/ Tue, 24 Sep 2019 08:20:07 -0400 Aegir DevOps: Deployment Workflows for Drupal Sites on Consensus Enterprises Blog published Tue, 24 Sep 2019 08:20:07 -0400 Aegir is often seen as a stand-alone application lifecycle management (ALM) system for hosting and managing Drupal sites. In the enterprise context, however, it’s necessary to provide mutiple deployment environments for quality assurance (QA), development or other purposes. Aegir trivializes this process by allowing sites to easily be copied from one environment to another in a point-and-click fashion from the Web front-end, eliminating the need for command-line DevOps tasks, which it was … Aegir is often seen as a stand-alone application lifecycle management (ALM) system for hosting and managing Drupal sites. In the enterprise context, however, it’s necessary to provide mutiple deployment environments for quality assurance (QA), development or other purposes. Aegir trivializes this process by allowing sites to easily be copied from one environment to another in a point-and-click fashion from the Web front-end, eliminating the need for command-line DevOps tasks, which it was designed to do. An Aegir instance needs to be installed in each environment. We would typically have three (3) of them: Development (Dev): While generally reserved for integration testing, it is sometimes also used for development (e.g. when local environments cannot be used by developers or there are a small number of them). Staging: Used for QA purposes. Designed to be a virtual clone of Production to ensure that tagged releases operate the same way as they would there, before being made live. Production (Prod): The live environment visible to the public or the target audience, and the authoritative source for data. (While outside the scope of this article, local development environments can be set up as well. See Try Aegir now with the new Dev VM for details.) To install Aegir in each of these, follow the installation instructions. For larger deployments, common architectures for Staging and Prod would include features such as: Separate Web and database servers Multiple Web and database servers Load balancers Caching/HTTPS proxies Separate partitions for (external) storage of: The Aegir file system (/var/aegir) Site backups (/var/aegir/backups) Database storage (/var/lib/mysql) etc. As these are all out of scope for the purposes of this article, I’ll save these discussions for the future, and assume we’re working with default installations. To enable inter-environment communication, we must perform the following series of tasks on each Aegir VM as part of the initial set-up, which only needs to be done once. The back-ends of each instance must be able to communicate. For that we use the secure SSH protocol. As stated on Wikipedia: SSH is important in cloud computing to solve connectivity problems, avoiding the security issues of exposing a cloud-based virtual machine directly on the Internet. An SSH tunnel can provide a secure path over the Internet, through a firewall to a virtual machine. Steps to enable SSH communication: SSH into the VM. ssh ENVIRONMENT.aegir.example.com Become the Aegir user. sudo -sHu aegir Generate an SSH key. (If you’ve done this already to access a private Git repository, you can skip this step.) ssh-keygen -t rsa -b 4096 -C "ORGANIZATION Aegir ENVIRONMENT" For every other environment from where you’d like to fetch sites: Add the generated public key (~/.ssh/id_rsa.pub) to the whitelist for the Aegir user on the other VM so that the original instance can connect to this target. ssh OTHER_ENVIRONMENT.aegir.example.com sudo -sHu aegir vi ~/.ssh/authorized_keys exit Back on the original VM, allow connections to the target VM. sudo -sHu aegir ssh OTHER_ENVIRONMENT.aegir.example.com Answer affirmatively when asked to confirm the host (after verifying the fingerprint, etc.). These steps will tell Aegir about the other Aegir servers whose sites can be imported. On Aegir’s front-end Web UI, the “hostmaster” site, enable remote site imports by navigating to Administration » Hosting » Advanced, and check the Remote import box. Save the form. (This enables the Aegir Hosting Remote Import module.) For every other server you’d like to add, do the following: Navigate to the Servers tab, and click on the Add server link. For the Server hostname, enter the hostname of the other Aegir server (e.g. staging.aegir.example.com) Click the Remote import vertical tab, check Remote hostmaster, and then enter aegir for the Remote user. For the Human-readable name, you can enter something like Foo's Staging Aegir (assuming the Staging instance). You can generally ignore the IP addresses section. Hit the Save button. Wait for the server verification to complete successfully. All of the one-time command-line tasks are now done. You or your users can now use the Web UI to shuffle site data between environments. Whenever necessary, this point-and-click process can be used to deploy sites from one Aegir environment to another. It’s actually a pull method as the destination Aegir instance imports a site from the source. Reasons to do this include: The initial deployment of a development site from Dev to Prod. Refreshing Dev and Staging sites from Prod. Steps: If you’d like to install the site onto a new platform that’s not yet available, create the platform first. Navigate to the Servers tab. Click on the server hosting the site you’d like to import. Click on the Import remote sites link. Follow the prompts. Wait for the batch job, Import and Verify tasks to complete. Enable the imported site by hitting the Run button on the Enable task. The imported site is now ready for use! https://consensus.enterprises/blog/try-aegir-now-with-the-new-dev-vm/ Mon, 09 Sep 2019 00:00:01 -0400 Try Aegir now with the new Dev VM on Consensus Enterprises Blog published Mon, 09 Sep 2019 00:00:01 -0400 Have you been looking for a self-hosted solution for hosting and managing Drupal sites? Would you like be able able to upgrade all of your sites at once with a single button click? Are you tired of dealing with all of the proprietary Drupal hosting providers that won’t let you customize your set-up? Wouldn’t it be nice if all of your sites had free automatically-updating HTTPS certificates? You probably know that Aegir can do all of this, but it’s now trivial to set up a temporary trial instance … Have you been looking for a self-hosted solution for hosting and managing Drupal sites? Would you like be able able to upgrade all of your sites at once with a single button click? Are you tired of dealing with all of the proprietary Drupal hosting providers that won’t let you customize your set-up? Wouldn’t it be nice if all of your sites had free automatically-updating HTTPS certificates? You probably know that Aegir can do all of this, but it’s now trivial to set up a temporary trial instance to see how it works. The new Aegir Development VM makes this possible. Throughout Aegir’s history, we’ve had several projects striving to achieve the same goal. They’re listed in the Contributed Projects section of the documentation. Aegir Up was based on a VirtualBox virtual machine (VM), managed by Vagrant and provisioned with Puppet. It was superseded by Valkyrie (see below). Aegir Development Environment took a completely different approach using Docker. It assembles all of the services (each one in a container, e.g. the MySQL database) into a system managed by Docker Compose. While this is a novel approach, it’s not necessary to have multiple containers to get a basic Aegir instance up and running. Valkyrie was similar to Aegir Up, but provisioning moved from Puppet to Ansible. Valkyrie also made extensive use of custom Drush commands to simplify development. Its focus was more on developing Drupal sites than on developing Aegir. Now that we have Lando, it’s no longer necessary to include this type of functionality. It was superseded by the now current Aegir Development VM. Like Valkyrie, the Aegir Development VM is based on a VirtualBox VM (but that’s not the only option; see below) managed with Vagrant and provisioned with Ansible. However, it doesn’t rely on custom Drush commands. The Aegir Development VM configuration is very easy to customize as Ansible variables are used throughout. For example, if you’d like to use Nginx instead of Apache, simply replace: aegir_http_service_type: apache …with: aegir_http_service_type: nginx …or override using the command line. You can also install and enable additional Aegir modules from the available set. For those folks with older hardware who are unable to spare extra gigabytes (GB) for VMs, it’s possible to set up the VM remotely. While the default amount of RAM necessary is 1 GB, 2 GB would be better for any serious work, and 4 GB is necessary if creating platforms directly from Packagist. Support for DigitalOcean is included, but other IaaS providers (e.g. OpenStack) can be added later. Patches welcome! While Aegir can quickly be installed with a small number of commands in the Quick Start Guide, that process requires an FQDN, usually something like aegir.example.com (which requires global DNS configuration). That is not the case with the Dev VM, which assumes aegir.local by default. You can use it for Aegir development as well as trying Aegir! Unlike the default set-up provisioned by the Quick Start Guide, which would require additional configuration, the individual components (e.g. Hosting, Provision, etc.) are cloned repositories making it easy to create patches (and for module maintainers: push changes upstream). We’ve recently updated the project so that an up-to-date VM is being used, and it’s now ready for general use. Please go ahead and try it. If you run into any problems, feel free to create issues on the issue board and/or submit merge requests.