https://consensus.enterprises/tags/celery/ Recent content tagged 'Celery' on the Consensus Enterprises Blog. Hugo -- gohugo.io en-CA info@consensus.enterprises (The Consensus Team) tech@consensus.enterprises (Consensus Infrastructure) Copyright 2025 Consensus Enterprises International Inc. Tue, 06 Jun 2023 09:00:00 -0400 https://consensus.enterprises/images/consensus-blog-banner.png https://consensus.enterprises/tags/celery/ https://consensus.enterprises/blog/aegir5-feature-parity/ Tue, 06 Jun 2023 09:00:00 -0400 Aegir5: Feature parity between Aegir3 and Aegir5 on Consensus Enterprises Blog published Tue, 06 Jun 2023 09:00:00 -0400 In previous posts, we’ve covered our Kubernetes framework for an alternative back-end to Aegir5, as well as the front-end Tasks and Operations and Clusters, Projects, Releases, and Environments. We also discussed the Queue architecture that ties the front- and back-ends together. This time, let’s consider the planned feature parity between Aegir 3 and Aegir 5. In Aegir3, Servers only have a “verify” task. This ensures that the front end can connect to the server over SSH. The new model will … In previous posts, we’ve covered our Kubernetes framework for an alternative back-end to Aegir5, as well as the front-end Tasks and Operations and Clusters, Projects, Releases, and Environments. We also discussed the Queue architecture that ties the front- and back-ends together. This time, let’s consider the planned feature parity between Aegir 3 and Aegir 5. In Aegir3, Servers only have a “verify” task. This ensures that the front end can connect to the server over SSH. The new model will introduce the analogous Cluster entity. This will work similarly, ensuring that the front-end can connect to the Kubernetes cluster. Platforms, for their part, generally have two tasks in Aegir 3: verify, and migrate sites. The Verify task largely is responsible for deploying the platform, as well as ensuring proper file permissions and so forth. Aegir5 represents a notable departure in this area. We’ve introduced two different entities to represent different aspects of what a Platform does in Aegir3. First, we have Projects, which are effectively a codebase- a git repository somewhere with an application that’s under development, evolving and improving. On top of that is a Release, which in the Kubernetes backend amounts to building a new container image. This represents a snapshot of the Project at a specific point in time: a version of the application, all its dependencies, and the runtime environment config needed to run it. In the short-term, our focus is on this cloud-native use-case, but the idea of Projects and Releases is applicable even in a more traditional VM-based scenario. At the infrastructure level, Clusters is the current target, but Aegir will support a more traditional Server as well, allowing Projects to be deployed onto varying infrastructure models. In Aegir3, Sites have a CRUD suite of tasks that include installing, verifying, upgrading and deleting. In Aegir5, this will be represented by Environments, into which we can deploy, update and delete the application. From the front end, this will look almost identical, but of course in the Kubernetes context these represent a running instance of the application: a container image of a particular Release, on a given Cluster. There is one more important distinction at this level. Aegir3 relies extensively on Drupal’s multisite capabilities. With Aegir5, we are setting aside that functionality almost entirely. Instead, we treat each site as standalone. This allows for support of large sites with complex development workflows. However, we can also track which sites use a given image. As a result we can (with a single operation) trigger migrations/updates for all the instances that are using that image. Aegir3 relies heavily on a bespoke task queue. As described previously, these Tasks have become Operations in Aegir5. They already have a log of the backend output. Operations in turn are composed of Tasks, which in Aegir5 are more granular, configurable steps in a larger whole. For example, writing a virtual host configuration, or setting up a HTTPS certificate. As discussed, we’ve modernized the task queue itself to use Celery, which is built atop RabbitMQ. This stack replaces the bespoke implementation and unlocks a more distributed “control plane” for the Aegir system. In addition to the task queue, Aegir3 also includes queues for running cron, taking scheduled backups, and so forth. Celery is certainly capable of implementing such queues. However, these are not an immediate priority. One feature from Aegir3 that we would like to incorporate is the ability to retry failed tasks. However, this may not make sense in a Kubernetes context. Backup and restore are important tasks for Aegir. In the long run, Aegir5’s Kubernetes backend will probably want to do volume snapshots using a tool like Valero. However, to keep things simple to start, we will operate similarly to how we do this in Aegir3, using an SQL dump and file tarball, a model which has served us well all these years. Cloning a site, in the long run, will be a composition of deploying from a snapshot/backup, taken immediately. Here again, the composition of this kind of operation becomes very similar to Aegir3. However, we can also simply deploy a new site, and then synchronize the database and files from the source site. Disabling a site will be a matter of running a job that rewrites the vhost to point to a static page, similarly to how Aegir3 does it now. As previously discussed in the Frontend Low-level Architecture post, the password reset functionality is much improved, by simply providing a “login to site” button. URL aliases and HTTPS capabilities are both already handled within the Kubernetes backend via the ingress and certificate manager services. These will be exposed via the Drupal UI as configuration options when creating an Environment to deploy a Release. The fundamental concepts of Aegir3 map quite readily onto the new Aegir5 architecture. There are some functional changes to modernize the overall framework, but from an end-user standpoint, the user experience (UX) of working with an Aegir5 system should feel quite familiar. The mapping described here shows how the new back-end and queue components integrate into this UX structure. In our next post, we’ll lay out our plan to move ahead and implement the missing pieces to get a fully functional MVP Aegir5 system. We are excited to be talking about this effort more publicly, and even more so to see more engagement with the Aegir5 project from the wider community. Let us know what you think and how you want to get involved! https://consensus.enterprises/blog/aegir5-pitchburgh/ Wed, 31 May 2023 09:00:00 -0300 Aegir5 + Pitch-burgh on Consensus Enterprises Blog published Wed, 31 May 2023 09:00:00 -0300 See our recent Aegir5 Roadmapping series for more background! Recently, Dries announced an exciting event at the upcoming DrupalCon Pittsburgh: Shark Tank meets Drupal: pitch your best innovation ideas/): Announcing “Pitch-burgh”, an innovation contest at DrupalCon Pittsburgh, where members of the Drupal community can pitch their ideas to receive funding. […] The entrepreneurs give short 2-3 minute presentations in the hopes of securing funding for their idea. Since we’re actively pursuing … See our recent Aegir5 Roadmapping series for more background! Recently, Dries announced an exciting event at the upcoming DrupalCon Pittsburgh: Shark Tank meets Drupal: pitch your best innovation ideas/): Announcing “Pitch-burgh”, an innovation contest at DrupalCon Pittsburgh, where members of the Drupal community can pitch their ideas to receive funding. […] The entrepreneurs give short 2-3 minute presentations in the hopes of securing funding for their idea. Since we’re actively pursuing partners and funding for Aegir5 development, this seemed like it could be a good opportunity. If nothing else, it’d give us a chance to develop a succinct message to describe this initiative. We’re pretty happy with the results. We hope you like it too (and read on below for more about the making of, and intention behind our pitch): Bridging to the Future: A plan to bring Aegir 5 to the Drupal community from Consensus Enterprises on Vimeo. Description: Consensus Enterprises is seeking funds to match its own investment in building Aegir 5: an application-agnostic, modern, and open-source community-owned hosting tool that works with Drupal 10. This will offer a migration path for Aegir 3 users, and will also enable management of non-Drupal sites and applications with entire software stacks. This funding will help us reach an alpha MVP more quickly, opening the door for other contributors in the community to also start developing the application. We had recently started working with an absolutely fantastic marketing consultant, Zack (storypanda.ca). When we asked him about it, he was quick to recommend Rob, with whom he’d collaborated successfully on previous projects. Rob (robspence.tv) is an amazing filmmaker and, interestingly, also a real-life cyborg. They both quickly agreed to help, so we started figuring out what we wanted to say. Zack led some initial brainstorming, and gave us some homework in the form of a questionnaire to help the Consensus-Aegir team to articulate our vision for the project. In the meantime, Rob had us scout locations within our respective homes. He also prompted us to start gathering some visual elements to incorporate into the video, so that we wouldn’t just be talking heads. Zack came up with a draft script that touched on all the important topics we’d come up with. After a round of revisions, we were ready to shoot. None of us felt particularly comfortable in front of the cameras, even if they were just the ones in our phones. Rob came through again, putting us at ease with witty banter, and confidence earned over decades of shooting and editing videos. With the shooting done, Rob put together a rough-cut. This was eye-opening, as we could finally get a sense of what the final product would look like. Several edits later, based on feedback from the team and Zack, we had our final product. To be frank, we’ve found it challenging to distill a succinct message in terms of our vision of Aegir, what it can be for the community, and what we want to do with it. “Here’s what we’re proposing…” Christopher begins, about 30 seconds into the video. He goes on to describe the urgent need to sunset Aegir3 and replace it with Aegir5. We’ve been cultivating this for several years, but we recognize the need to re-open the Aegir project to community participation and contribution. In the video, we identify a few deliverables, which focus on releasing Aegir5 back into the Drupal community at large. The core of this milestone is to wire up the functional components we’ve built into a coherent minimal viable product (MVP). This will address the immediate use-case we’re seeing in our own work, while allowing the flexibility to expand functionality to other cases. Fundamentally, the MVP will provide a functional Drupal app hosting platform which can replace Aegir3, ideally before Drupal 7 goes EOL. In the Alpha MVP stage, the goal is to release a stable “core” of Aegir5, a developer-ready foundation on which to build further. Improving the developer experience (DX) in Aegir5 is a key design goal, since we want the system to support a wide array of features we can’t build all on our own. To that end, we plan to review, update, and improve our documentation, continue building out example modules which demonstrate how to leverage the underlying tools, as well as a comprehensive test suite. We view testing as both a quality control, guarding against regressions and other breakage, but also a means to illustrate how the system behaves. We’re working in the open on this, so feel free to check out the milestone details and engage with us on the GitLab project anytime :) Once we complete the Alpha MVP milestone, we will start incorporating feedback from the existing Aegir developer community (shout-outs to Omega8.cc, DevShop, and InitFour.nl!) and beyond. We’ve already had some great conversations with existing Aegir3 users, and we plan to evolve the Beta MVP (and subsequent) milestone(s) based on the interest and feedback we gather in this process. Since the original Hostmaster evolved into Aegir over a decade ago, many commercial Drupal hosting providers (Acquia, Pantheon, Platform.sh, etc) have come to the fore and serve a valuable role in the ongoing growth of Drupal on the web. As strong open source advocates, we believe it’s important to have an open, free alternative to commercial providers where we can address the additional use-cases in this space (eg. self-hosters, SaaS owners, etc.) and to help keep the Drupal devOps community open and growing. Watch this space for an upcoming series to talk in more detail about these use-cases and how we imagine the ongoing evolution of Aegir5 :) Consensus plans to continue being a community steward for Aegir, as we feel passionate about it’s value both to ourselves and as a community. We look forward to wider participation in this effort. Please get in touch if you’re interested in supporting Aegir5! Wish us luck at DrupalCon Pitch-burg! https://consensus.enterprises/blog/aegir5-queue-architecture/ Wed, 26 Apr 2023 09:00:00 -0500 Aegir5: Queue Architecture on Consensus Enterprises Blog published Wed, 26 Apr 2023 09:00:00 -0500 Previously in this series, we looked at the Aegir5 front-end interface architecture, as well as the lower level entities, Tasks and Operations that provide building blocks. As mentioned in the first part of the series, our most recent work on Aegir5 itself has been reworking the queue system. In this post, we explore this topic in more detail. The Aegir5 queue is implemented using Celery, which is a full-featured Python-based task queue, built atop RabbitMQ. Initially we built dispatcherd which … Previously in this series, we looked at the Aegir5 front-end interface architecture, as well as the lower level entities, Tasks and Operations that provide building blocks. As mentioned in the first part of the series, our most recent work on Aegir5 itself has been reworking the queue system. In this post, we explore this topic in more detail. The Aegir5 queue is implemented using Celery, which is a full-featured Python-based task queue, built atop RabbitMQ. Initially we built dispatcherd which operates as the back-end queue worker. For a variety of reasons that we’ll get into later, we also added relayd to run on the front end. dispatcherd is the Aegir5 equivalent of Drush Provision. It is responsible for writing configuration files and then running the commands that will instantiate whatever the configurations represent. dispatcherd runs commands such as ansible playbook. It then gathers the terminal output and sends it back to the front-end. Initially, this was done using an HTTP end-point, which would update the log record of the relevant operation. This worked reasonably well, but it required HTTP access from dispatcherd to the front end. This is where relayd comes into play. By hitting the front end with updates from the terminal output, we had to bootstrap the web UI repeatedly, putting unnecessary load on the front end. So we built an alternative solution: relayd running as a queue worker on the front end. This front end worker is very lightweight, picking up data and passing it on to an aegir:console command that can be more efficient at injecting the log results from the back end. Once we had added relayd, we realized it presented another opportunity. Up to this point, the front end had been posting directly into the queue for dispatcherd. However, this required the front end to have credentials to the Celery queue. Since the intention was for relayd to run in the same context as the front end, we could instead simply have the front end communicate with relayd securely without providing credentials to the back-end. In addition to a more loose coupling of all the components, this represents an improved security posture for Aegir5 overall. By removing queue credentials from the front-end UI, we reduce the risk of a vulnerability in the Drupal site leading to a compromise of the back-end infrasructure. Having built and tested the queue architecture fairly thoroughly we are poised to tie everything together. In the next post, we explore the integration of our Kubernetes framework with the queue mechanism and existing front-end UI to create a fully functional (albeit MVP) Aegir5 implementation. https://consensus.enterprises/blog/aegir5-frontend-ui/ Tue, 18 Apr 2023 09:00:00 -0500 Aegir5: Front-end UI architecture on Consensus Enterprises Blog published Tue, 18 Apr 2023 09:00:00 -0500 In our previous post, we looked at the Tasks and Operations which form the building blocks for the user interface in Aegir5. Here we’ll look at the additional entities required to support the Kubernetes-based backend framework. It is worth noting that Aegir has always had a tension between Developer and SysAdmin use-cases. We’ll cover this in more depth in a later post. For the moment, we’re focused on the Developer use-case. One thing to bear in mind is that Kubernetes operates quite … In our previous post, we looked at the Tasks and Operations which form the building blocks for the user interface in Aegir5. Here we’ll look at the additional entities required to support the Kubernetes-based backend framework. It is worth noting that Aegir has always had a tension between Developer and SysAdmin use-cases. We’ll cover this in more depth in a later post. For the moment, we’re focused on the Developer use-case. One thing to bear in mind is that Kubernetes operates quite differently than configuring and operating Nginx and MySQL directly, as we do with Aegir 3 Provision. As a result, the types of entities we are working with are going to be somewhat different. However, they will largely map to existing concepts in Aegir 3. As seen in the diagram from the Kubernetes backend post (reproduced below), the core entities are dealing with are Clusters, Projects, Releases and Environments that we’re managing. N.B. The terms below are how we’re currently describing the high-level components. Our goal here is to make these as natural and clear as possible. This may evolve over time, especially if we have to continuously explain what any of them mean (i.e. “Platform”). Clusters, in this case, map fairly well to Servers. We intend to support provisioning of Clusters in the future. For the time-being, we will simply be pointing at existing Clusters to which we will want to deploy our applications. Projects represent the codebase at a more abstract level. Whereas Releases (see below) are fully instantiated codebases, Projects represent the source of the codebase itself; usually a Git repository. We may introduce a similar entity type, Applications, down the road. These would serve a similar purpose, but optimized for “cattle” in the SysAdmin use-case. Devshop and the Aegir Distributions module provide similar functionality in Aegir 3. Releases are basically the equivalent of Platforms in Aegir 3. They capture the runtime environment, with a specific version of our codebase deployed and all dependencies downloaded. In Kubernetes, these are basically container images. Sharing a common codebase across multiple sites can be achieved by leveraging the same Release. However, unlike Platforms, we are not intending to support Drupal’s multisite capability in Aegir5. Multisite adds a lot of complexity, especially when running in a containerized environment. It also introduces multiple security and stability concerns. Environments become the equivalent of Sites in Aegir 3. They capture the specific configuration required to run an instance of a specific Release of the Project (or Application), along with the storage required for both the database and any uploaded files. These are what we believe to be the basic building blocks that we need to wire up to the Drumkit Kubernetes backend. We’ve tried not to bake in too tight of a coupling to the Kubernetes backend. They ought to be re-usable for the Ansible backend, as well. Our medium-term goal is to build a robust, reliable and flexible continuous delivery system. Some of this will likely require much more opinionated approaches. We want to keep the basic components minimally opinionated, so that they can be composed into more elaborate workflows to satisfy different use-cases (eg. SaaS operations). So far in this series we’ve looked at the Kubernetes framework we’ve built which will form the new backend for Aegir5. We’ve also explored the Drupal content architecture that forms the front-end. In the next post we’ll look at the Queue architecture which ties the front-end and back-end together before moving on to our Roadmap for bringing the whole thing together. https://consensus.enterprises/blog/aegir5-frontend-low-level/ Thu, 06 Apr 2023 09:00:00 -0500 Aegir5: Front-end low-level architecture on Consensus Enterprises Blog published Thu, 06 Apr 2023 09:00:00 -0500 In our previous post, we talked about our recent client work building a Kubernetes-based system for hosting web applications. We’ve defined a general framework to support our development and production hosting workflows, and recognized this as a solid basis for an alternate backend to plug in to the existing Aegir5 front-end. Today we’ll take a look at the Drupal architecture underlying that front-end. In Aegir5, the building blocks consist of Task and Operation entities. Tasks represent … In our previous post, we talked about our recent client work building a Kubernetes-based system for hosting web applications. We’ve defined a general framework to support our development and production hosting workflows, and recognized this as a solid basis for an alternate backend to plug in to the existing Aegir5 front-end. Today we’ll take a look at the Drupal architecture underlying that front-end. In Aegir5, the building blocks consist of Task and Operation entities. Tasks represent relatively low-level reusable functionality. For example, we might have a Task for the Ansible back-end that is responsible for writing an Nginx vhost. We might then bundle that Task, along with others, in order to compose an “Install Drupal” Operation. Since the need to write a vhost is pretty common for web applications, that same Task might also be bundled into an “Install Joomla” Operation. Operations represent the actions that a user might want to trigger. Operations are configured to contain one or more Tasks. Each Task is a standalone fieldable entity. As a result each Task is responsible for gathering whatever data it needs to run its back-end tasks. The Operation includes these Tasks as inline entities. When we trigger an Operation, we are presented with a form that includes all the fields from the various Tasks associated with that Operation. These values, across all of the related Tasks, are marshalled and sent to the back-end. Logs are kept at the Operation level and so the output from the back-end gets piped back into a field on the Operation. In addition to the terminal output from the back-end, we also include the return code of the operation, so that we can record whether it succeeded or not. Another type of task is what we call an “action” task. These do not get bundled into Operations, but rather happen within a given page request. For example, instead of explicitly triggering a “password reset” task, as we do in Aegir 3, we can have a task dispatched when we click a “Log into site” button. This task runs a drush uli (via the Celery task queue), and relays the result directly back to the front-end, almost immediately. This can in turn redirect us directly to that URL. The result is that we are logged into the site without having to wait for a fully logged Operation to run. In the same way that Operations are made up of Tasks via inline entities, other entities such as Sites and Platforms are themselves made up of Operations embedded as inline entities. When we create a Platform entity, we instantiate all of its Operations and related Tasks. Among other things, this allows us to reuse Tasks between Operations. For example, when, taking a backup, we will want to record the file path to the tarball. This data then becomes available to the restore task that may come later. This case also illustrates the need for the back-end to be able to write data to the front-end. This is another function of relayd in the queue architecture. (see the upcoming post on this for more details). In addition to writing logs to Operation entities, it can write data to any field within any entity on the front-end. The sequence diagram below illustrates the flow from an Operation being triggered to run, through placing a task on the queue, running the backend executor (currently Ansible, soon Kubernetes), and getting feedback returned to the frontend. This architecture largely mirrors how Aegir 3 works. However, it provides a more flexible, robust and scalable solution. In our next post, we’ll take a look at the higher-level UI architecture we plan to implement to represent the key components in our Kubernetes hosting framework: Clusters, Projects, Releases, and Environments. https://consensus.enterprises/blog/aegir5-is-happening/ Sat, 25 Feb 2023 09:00:00 -0500 Aegir5 Development is Happening! on Consensus Enterprises Blog published Sat, 25 Feb 2023 09:00:00 -0500 Aegir5 development is happening! We (Consensus) have been making steady progress on it over the last few years and are looking to kick off a new burst of focused development. Here’s a summary of progress that has been made so far and how you can contribute. First off, as you’re probably aware, Aegir5 is a complete re-write of Aegir. We are intending to build on all the great aspects of Aegir, while freeing ourselves from a codebase that is rooted in PHP 4. We’re using D9+ alongside modern … Aegir5 development is happening! We (Consensus) have been making steady progress on it over the last few years and are looking to kick off a new burst of focused development. Here’s a summary of progress that has been made so far and how you can contribute. First off, as you’re probably aware, Aegir5 is a complete re-write of Aegir. We are intending to build on all the great aspects of Aegir, while freeing ourselves from a codebase that is rooted in PHP 4. We’re using D9+ alongside modern middleware and back-end components. The front-end (Drupal) and the queue system (based on Celery) are working fairly well. We’ve mostly reproduced the Aegir 3 behaviour, and it has a similar look and feel. This is the most recent work that’s been done directly in the Aegir5 repo, and dates back about a year, or more. (See for example aegir#94.) It’s suffering from some bitrot, so we’ll need to do a bit of cleanup. Bear in mind that we’re self-funding this project, so we aren’t currently able to dedicate much time and budget to it directly. However, we have had a number of contracts, both current and in the recent past, where we’ve been able to evolve some of the middleware and back-end components. For example, we recently worked with the Drupal Association to build Rugged, a TUF server implementation to secure the Drupal Composer repository. The Drupal Infrastructure team is in the process of beta testing this functionality on drupal.org, and is very much in line with the Aegir project’s goals of making updates easy and secure. More directly related to Aegir5, Rugged uses Celery extensively, and started out building from the components we’re using in Aegir5. We’ve developed them much further within the context of that project, and plan to port those improvements back to Aegir5. Likewise, our initial plan was to build the Aegir5 back-end using Ansible. While we have some basic playbook elements in Aegir5 currently, we’ve had a number of clients recently ask us to build fully automated Kubernetes-based hosting systems. Our first prototype is hosting a Django/React app, and is working well in production. We’ve been hosting our Kubernetes clusters on OpenStack, since we’ve been striving for a fully FLOSS stack as part of our overall goals for Aegir5. However, a number of our larger institutional clients are restricted to using one of the major cloud vendors. So we’ve been architecting our automation to support any Kubernetes cluster. We’re currently refining these components for a Drupal upgrade project for a department of the Government of Canada. The Drupal-on-Kubernetes piece was mostly completed a couple weeks ago. We’re continuing to enhance and stabilize it. Next, we’re planning to integrate something along those lines as the default back-end for Aegir5. We do not have a timeline for the integration of these components into Aegir5, mostly due to the aforementioned time/budget constraints. However, we are eager to proceed with that work. We have had a handful of organizations express interest in possibly funding some of this upcoming work, some of whom are currently hosting several hundred Drupal sites each on Aegir 3. If you know of someone who might be interested in helping to fund this work, please let us know. We’re undertaking to update the Aegir5 roadmap to reflect the progress we’ve made, and to outline next steps. Ideally we’ll be able to get a detailed enough break-down to be able to broadly estimate remaining effort for an alpha release. Hopefully this gives a good picture of the current state of Aegir5 development. We’ll be publishing some subsequent posts in the coming days to describe in more detail our Drupal-on-Kubernetes architecture, as well as how we aim to integrate the existing frontend and queue system work to achieve an MVP Aegir5. If you have any questions, don’t hesitate to ask. Aside from our Contact form, you can find us on LinkedIn or Matrix, and always eager to talk about Aegir!