3 minute read Published: Author: Christopher Gervais
Aegir , Aegir5 , Automation , Celery , DevOps , Drupal Planet , Kubernetes , OpenStack , Rugged



Previously in this series, we looked at the Aegir5 front-end interface architecture, as well as the lower level entities, Tasks and Operations that provide building blocks.

As mentioned in the first part of the series, our most recent work on Aegir5 itself has been reworking the queue system. In this post, we explore this topic in more detail.

The Aegir5 queue is implemented using Celery, which is a full-featured Python-based task queue, built atop RabbitMQ. Initially we built dispatcherd which operates as the back-end queue worker. For a variety of reasons that we’ll get into later, we also added relayd to run on the front end.

Aegi5 Queue Architecture diagram

dispatcherd is the Aegir5 equivalent of Drush Provision. It is responsible for writing configuration files and then running the commands that will instantiate whatever the configurations represent.

dispatcherd runs commands such as ansible playbook. It then gathers the terminal output and sends it back to the front-end. Initially, this was done using an HTTP end-point, which would update the log record of the relevant operation. This worked reasonably well, but it required HTTP access from dispatcherd to the front end.

This is where relayd comes into play. By hitting the front end with updates from the terminal output, we had to bootstrap the web UI repeatedly, putting unnecessary load on the front end. So we built an alternative solution: relayd running as a queue worker on the front end. This front end worker is very lightweight, picking up data and passing it on to an aegir:console command that can be more efficient at injecting the log results from the back end.

Once we had added relayd, we realized it presented another opportunity. Up to this point, the front end had been posting directly into the queue for dispatcherd. However, this required the front end to have credentials to the Celery queue. Since the intention was for relayd to run in the same context as the front end, we could instead simply have the front end communicate with relayd securely without providing credentials to the back-end.

In addition to a more loose coupling of all the components, this represents an improved security posture for Aegir5 overall. By removing queue credentials from the front-end UI, we reduce the risk of a vulnerability in the Drupal site leading to a compromise of the back-end infrasructure.

Having built and tested the queue architecture fairly thoroughly we are poised to tie everything together. In the next post, we explore the integration of our Kubernetes framework with the queue mechanism and existing front-end UI to create a fully functional (albeit MVP) Aegir5 implementation.


The article Aegir5: Queue Architecture first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!