https://consensus.enterprises/tags/rugged/ Recent content tagged 'Rugged' on the Consensus Enterprises Blog. Hugo -- gohugo.io en-CA info@consensus.enterprises (The Consensus Team) tech@consensus.enterprises (Consensus Infrastructure) Copyright 2025 Consensus Enterprises International Inc. Mon, 19 Aug 2024 15:25:00 -0400 https://consensus.enterprises/images/consensus-blog-banner.png https://consensus.enterprises/tags/rugged/ https://consensus.enterprises/blog/starshot-moving-drupal-towards-a-product-platform/ Mon, 19 Aug 2024 15:25:00 -0400 Starshot: Moving Drupal Towards a Product Platform on Consensus Enterprises Blog published Mon, 19 Aug 2024 15:25:00 -0400 In the world of Drupal, the terms “product” and “framework” represent two different approaches to how the platform can be used and who it serves. A product in the Drupal sense refers to a ready-to-use solution that allows users to build and manage websites with minimal technical knowledge. It’s about offering a polished, user-friendly experience where the focus is on enabling non-developers — such as content editors or small organizations — to easily create and maintain a web presence without … In the world of Drupal, the terms “product” and “framework” represent two different approaches to how the platform can be used and who it serves. A product in the Drupal sense refers to a ready-to-use solution that allows users to build and manage websites with minimal technical knowledge. It’s about offering a polished, user-friendly experience where the focus is on enabling non-developers — such as content editors or small organizations — to easily create and maintain a web presence without needing to understand the underlying code. The goal is to make Drupal useful and approachable out-of-the-box, with intuitive interfaces and pre-configured options. On the other hand, a framework is more of a toolkit for developers. It provides the building blocks to create highly customized and complex web applications. A framework is designed with flexibility in mind, allowing developers to build everything from basic sites to intricate systems tailored to specific needs. However, this approach often requires a deeper understanding of web development and can be more resource-intensive to implement and manage. Earlier in Drupal’s history, a common refrain was “Kill the Webmaster”. This slogan positioned Drupal as a product, aiming to create an out-of-the-box experience that would lower the barrier for entry for non-technical users. The idea was to incorporate web development best-practices, making it possible for more people and small organizations to publish their own websites without needing to be experts in web technologies. Around the point where Drupal 8 development began, a countervailing trend arose seeking to strip Drupal core down to be more of a framework; known as “small core” at the time. There were multiple reasons for this, including core developer burnout, maintaining such a large codebase. This shift was driven partly by the needs of developers and larger organizations to build complex, custom solutions. While this made Drupal a powerful tool for those with the skills to use it, it also meant that the product-like simplicity envisioned earlier was less prioritized. This evolution left some smaller organizations feeling alienated, as they found it more challenging to use Drupal without technical expertise. The Internet has evolved a lot since the introduction of Drupal 8. But then, so has Drupal itself. Part of the development of Drupal 8 included leveraging the test suite (largely introduced with Drupal 7) to basically rip out and replace a lot of Drupal’s own internals, and make them more object-oriented. This, in turn, was intended to make the upgrade cycle from one major version to the next less painful. It is true that the upgrade process from Drupal 7 to 8 was a very significant lift, but, from Drupal 8 onward, upgrades have been rendered almost trivial! Today, with the recent release of Drupal 11, the vast majority of that movement away from that Drupal 7 world has occurred within Drupal itself, but a significant number of Drupal sites are still on Drupal 7. In parallel, we’ve seen an explosion in the popularity of comparable projects like Wordpress, which takes that product approach and runs with it. Now that Drupal core has evolved into the powerful framework it is today, the community is no longer faced with an either-or choice. We can, in fact, have our cake (Drupal core) and eat it (Starshot) too. Recently, Consensus colleagues Dan Friedman and Brian Sharpe had the chance to talk at DrupalCamp Ottawa about the promise of Starshot. Starshot represents a shift in the Framework-vs-Product conversation, bringing the best of Drupal’s framework into the product development arena. We see this as an opportunity for Drupal to say to those smaller organizations: “We’re going to build something that works for you, for the skillset that you actually have.” In short, “Kill the Webmaster” has become “Low Code/No Code”. We think this turn toward simplicity and product focus provides some very exciting opportunities. Automatic Updates will provide an upgrade process that doesn’t require an entire devops team to keep running. Behind the scenes, the Drupal infrastructure team is integrating software supply chain security into their packaging pipeline. The Drupal Association has sponsored the development of a Composer plugin and Rugged, the server-side component developed primarily by our colleague, Christopher Gervais. As a result, out-of-the-box Starshot (and Drupal 11) will provide users the confidence that the software they have requested is in fact the software they are receiving. Project Browser will allow users to manage installed modules and themes. Because this involves downloading new code into a running site, it has the potential to be insecure. Here at Consensus, we’re also developing Aegir 5, a self-hosting platform in which it will be relatively simple for the user to temporarily grant access for this task, then re-secure the site, from outside of Drupal itself. Other hosting providers will likely follow suit. Drupal core will be free to streamline the “framework” even further. Its modular approach allows us to selectively enable only the components we need, like choosing not to use Node when it’s unnecessary. This gives us the flexibility to build custom tools and configurations without relying on every part of the traditional Drupal components, tailoring the CMS precisely to our needs. Most of the previous items are “behind the scenes:” folks responsible for maintaining a site wouldn’t normally interact with most of them on a regular basis. Where things start to get really interesting is in seeing Drupal itself as a product development platform. Typically, a Drupal site is made up of: Drupal Core, providing base functionality; Community contributed modules, providing additional functionality; and, Themes, granting control over the site’s look and feel. Install profiles, which do all of the initial setup, get us to the point where we have a site that “does a particular thing” – they have always been the thing that we think of at Consensus when we think of a Drupal site being a particular application. But, there are a set of limitations on install profiles that have, historically, held Drupal back from being a product-oriented platform. For example, crucially, once you have installed a site using a certain profile, you can’t shift to a new one (via profiles alone – you need to involve migrations, or other mechanisms). So, if you offer a service that has Bronze, Silver and Gold packages, you can’t move a given client site from one “level” to the next via install profiles alone. They don’t, by themselves, provide a long-term maintenance or sequential migration strategy for any given web site “product.” In the course of various development projects at Consensus, we’ve had to come up with strategies for automating import/creation of various classes of data, including demo content, default content, test content, and so on. So we’re excited by Recipes that provide simplified content import, among other “low-code” (YAML only) interventions. This provides possibilities for getting past the limitations of install profiles. Another interesting inclusion in Starshot is the notion of a single-directory application, that further integrates 3rd party tools (javascript libraries, and so on) into a Drupal application, with minimal developer intervention. All in all, we see Starshot as a tremendously exciting and fascinating next step in Drupal’s evolution toward a product development platform. We look forward to supporting it, both through our ongoing work with the Drupal Association, and our continued work on Aegir 5. Drupal has evolved into a powerful platform for developing digital products, with tools like Starshot enhancing its versatility. Whether you need a straightforward website or a more complex application, Drupal’s open-source nature and flexibility make it a great option. At Consensus Enterprises, we’re passionate about open-source technology and committed to helping you achieve your goals. If you’re looking for a partner who values innovation and practicality, let’s talk about how we can use Drupal to make your vision a reality. Reach out to us today to get started. https://consensus.enterprises/blog/aegir5-feature-parity/ Tue, 06 Jun 2023 09:00:00 -0400 Aegir5: Feature parity between Aegir3 and Aegir5 on Consensus Enterprises Blog published Tue, 06 Jun 2023 09:00:00 -0400 In previous posts, we’ve covered our Kubernetes framework for an alternative back-end to Aegir5, as well as the front-end Tasks and Operations and Clusters, Projects, Releases, and Environments. We also discussed the Queue architecture that ties the front- and back-ends together. This time, let’s consider the planned feature parity between Aegir 3 and Aegir 5. In Aegir3, Servers only have a “verify” task. This ensures that the front end can connect to the server over SSH. The new model will … In previous posts, we’ve covered our Kubernetes framework for an alternative back-end to Aegir5, as well as the front-end Tasks and Operations and Clusters, Projects, Releases, and Environments. We also discussed the Queue architecture that ties the front- and back-ends together. This time, let’s consider the planned feature parity between Aegir 3 and Aegir 5. In Aegir3, Servers only have a “verify” task. This ensures that the front end can connect to the server over SSH. The new model will introduce the analogous Cluster entity. This will work similarly, ensuring that the front-end can connect to the Kubernetes cluster. Platforms, for their part, generally have two tasks in Aegir 3: verify, and migrate sites. The Verify task largely is responsible for deploying the platform, as well as ensuring proper file permissions and so forth. Aegir5 represents a notable departure in this area. We’ve introduced two different entities to represent different aspects of what a Platform does in Aegir3. First, we have Projects, which are effectively a codebase- a git repository somewhere with an application that’s under development, evolving and improving. On top of that is a Release, which in the Kubernetes backend amounts to building a new container image. This represents a snapshot of the Project at a specific point in time: a version of the application, all its dependencies, and the runtime environment config needed to run it. In the short-term, our focus is on this cloud-native use-case, but the idea of Projects and Releases is applicable even in a more traditional VM-based scenario. At the infrastructure level, Clusters is the current target, but Aegir will support a more traditional Server as well, allowing Projects to be deployed onto varying infrastructure models. In Aegir3, Sites have a CRUD suite of tasks that include installing, verifying, upgrading and deleting. In Aegir5, this will be represented by Environments, into which we can deploy, update and delete the application. From the front end, this will look almost identical, but of course in the Kubernetes context these represent a running instance of the application: a container image of a particular Release, on a given Cluster. There is one more important distinction at this level. Aegir3 relies extensively on Drupal’s multisite capabilities. With Aegir5, we are setting aside that functionality almost entirely. Instead, we treat each site as standalone. This allows for support of large sites with complex development workflows. However, we can also track which sites use a given image. As a result we can (with a single operation) trigger migrations/updates for all the instances that are using that image. Aegir3 relies heavily on a bespoke task queue. As described previously, these Tasks have become Operations in Aegir5. They already have a log of the backend output. Operations in turn are composed of Tasks, which in Aegir5 are more granular, configurable steps in a larger whole. For example, writing a virtual host configuration, or setting up a HTTPS certificate. As discussed, we’ve modernized the task queue itself to use Celery, which is built atop RabbitMQ. This stack replaces the bespoke implementation and unlocks a more distributed “control plane” for the Aegir system. In addition to the task queue, Aegir3 also includes queues for running cron, taking scheduled backups, and so forth. Celery is certainly capable of implementing such queues. However, these are not an immediate priority. One feature from Aegir3 that we would like to incorporate is the ability to retry failed tasks. However, this may not make sense in a Kubernetes context. Backup and restore are important tasks for Aegir. In the long run, Aegir5’s Kubernetes backend will probably want to do volume snapshots using a tool like Valero. However, to keep things simple to start, we will operate similarly to how we do this in Aegir3, using an SQL dump and file tarball, a model which has served us well all these years. Cloning a site, in the long run, will be a composition of deploying from a snapshot/backup, taken immediately. Here again, the composition of this kind of operation becomes very similar to Aegir3. However, we can also simply deploy a new site, and then synchronize the database and files from the source site. Disabling a site will be a matter of running a job that rewrites the vhost to point to a static page, similarly to how Aegir3 does it now. As previously discussed in the Frontend Low-level Architecture post, the password reset functionality is much improved, by simply providing a “login to site” button. URL aliases and HTTPS capabilities are both already handled within the Kubernetes backend via the ingress and certificate manager services. These will be exposed via the Drupal UI as configuration options when creating an Environment to deploy a Release. The fundamental concepts of Aegir3 map quite readily onto the new Aegir5 architecture. There are some functional changes to modernize the overall framework, but from an end-user standpoint, the user experience (UX) of working with an Aegir5 system should feel quite familiar. The mapping described here shows how the new back-end and queue components integrate into this UX structure. In our next post, we’ll lay out our plan to move ahead and implement the missing pieces to get a fully functional MVP Aegir5 system. We are excited to be talking about this effort more publicly, and even more so to see more engagement with the Aegir5 project from the wider community. Let us know what you think and how you want to get involved! https://consensus.enterprises/blog/aegir5-queue-architecture/ Wed, 26 Apr 2023 09:00:00 -0500 Aegir5: Queue Architecture on Consensus Enterprises Blog published Wed, 26 Apr 2023 09:00:00 -0500 Previously in this series, we looked at the Aegir5 front-end interface architecture, as well as the lower level entities, Tasks and Operations that provide building blocks. As mentioned in the first part of the series, our most recent work on Aegir5 itself has been reworking the queue system. In this post, we explore this topic in more detail. The Aegir5 queue is implemented using Celery, which is a full-featured Python-based task queue, built atop RabbitMQ. Initially we built dispatcherd which … Previously in this series, we looked at the Aegir5 front-end interface architecture, as well as the lower level entities, Tasks and Operations that provide building blocks. As mentioned in the first part of the series, our most recent work on Aegir5 itself has been reworking the queue system. In this post, we explore this topic in more detail. The Aegir5 queue is implemented using Celery, which is a full-featured Python-based task queue, built atop RabbitMQ. Initially we built dispatcherd which operates as the back-end queue worker. For a variety of reasons that we’ll get into later, we also added relayd to run on the front end. dispatcherd is the Aegir5 equivalent of Drush Provision. It is responsible for writing configuration files and then running the commands that will instantiate whatever the configurations represent. dispatcherd runs commands such as ansible playbook. It then gathers the terminal output and sends it back to the front-end. Initially, this was done using an HTTP end-point, which would update the log record of the relevant operation. This worked reasonably well, but it required HTTP access from dispatcherd to the front end. This is where relayd comes into play. By hitting the front end with updates from the terminal output, we had to bootstrap the web UI repeatedly, putting unnecessary load on the front end. So we built an alternative solution: relayd running as a queue worker on the front end. This front end worker is very lightweight, picking up data and passing it on to an aegir:console command that can be more efficient at injecting the log results from the back end. Once we had added relayd, we realized it presented another opportunity. Up to this point, the front end had been posting directly into the queue for dispatcherd. However, this required the front end to have credentials to the Celery queue. Since the intention was for relayd to run in the same context as the front end, we could instead simply have the front end communicate with relayd securely without providing credentials to the back-end. In addition to a more loose coupling of all the components, this represents an improved security posture for Aegir5 overall. By removing queue credentials from the front-end UI, we reduce the risk of a vulnerability in the Drupal site leading to a compromise of the back-end infrasructure. Having built and tested the queue architecture fairly thoroughly we are poised to tie everything together. In the next post, we explore the integration of our Kubernetes framework with the queue mechanism and existing front-end UI to create a fully functional (albeit MVP) Aegir5 implementation. https://consensus.enterprises/blog/aegir5-frontend-ui/ Tue, 18 Apr 2023 09:00:00 -0500 Aegir5: Front-end UI architecture on Consensus Enterprises Blog published Tue, 18 Apr 2023 09:00:00 -0500 In our previous post, we looked at the Tasks and Operations which form the building blocks for the user interface in Aegir5. Here we’ll look at the additional entities required to support the Kubernetes-based backend framework. It is worth noting that Aegir has always had a tension between Developer and SysAdmin use-cases. We’ll cover this in more depth in a later post. For the moment, we’re focused on the Developer use-case. One thing to bear in mind is that Kubernetes operates quite … In our previous post, we looked at the Tasks and Operations which form the building blocks for the user interface in Aegir5. Here we’ll look at the additional entities required to support the Kubernetes-based backend framework. It is worth noting that Aegir has always had a tension between Developer and SysAdmin use-cases. We’ll cover this in more depth in a later post. For the moment, we’re focused on the Developer use-case. One thing to bear in mind is that Kubernetes operates quite differently than configuring and operating Nginx and MySQL directly, as we do with Aegir 3 Provision. As a result, the types of entities we are working with are going to be somewhat different. However, they will largely map to existing concepts in Aegir 3. As seen in the diagram from the Kubernetes backend post (reproduced below), the core entities are dealing with are Clusters, Projects, Releases and Environments that we’re managing. N.B. The terms below are how we’re currently describing the high-level components. Our goal here is to make these as natural and clear as possible. This may evolve over time, especially if we have to continuously explain what any of them mean (i.e. “Platform”). Clusters, in this case, map fairly well to Servers. We intend to support provisioning of Clusters in the future. For the time-being, we will simply be pointing at existing Clusters to which we will want to deploy our applications. Projects represent the codebase at a more abstract level. Whereas Releases (see below) are fully instantiated codebases, Projects represent the source of the codebase itself; usually a Git repository. We may introduce a similar entity type, Applications, down the road. These would serve a similar purpose, but optimized for “cattle” in the SysAdmin use-case. Devshop and the Aegir Distributions module provide similar functionality in Aegir 3. Releases are basically the equivalent of Platforms in Aegir 3. They capture the runtime environment, with a specific version of our codebase deployed and all dependencies downloaded. In Kubernetes, these are basically container images. Sharing a common codebase across multiple sites can be achieved by leveraging the same Release. However, unlike Platforms, we are not intending to support Drupal’s multisite capability in Aegir5. Multisite adds a lot of complexity, especially when running in a containerized environment. It also introduces multiple security and stability concerns. Environments become the equivalent of Sites in Aegir 3. They capture the specific configuration required to run an instance of a specific Release of the Project (or Application), along with the storage required for both the database and any uploaded files. These are what we believe to be the basic building blocks that we need to wire up to the Drumkit Kubernetes backend. We’ve tried not to bake in too tight of a coupling to the Kubernetes backend. They ought to be re-usable for the Ansible backend, as well. Our medium-term goal is to build a robust, reliable and flexible continuous delivery system. Some of this will likely require much more opinionated approaches. We want to keep the basic components minimally opinionated, so that they can be composed into more elaborate workflows to satisfy different use-cases (eg. SaaS operations). So far in this series we’ve looked at the Kubernetes framework we’ve built which will form the new backend for Aegir5. We’ve also explored the Drupal content architecture that forms the front-end. In the next post we’ll look at the Queue architecture which ties the front-end and back-end together before moving on to our Roadmap for bringing the whole thing together. https://consensus.enterprises/blog/aegir5-frontend-low-level/ Thu, 06 Apr 2023 09:00:00 -0500 Aegir5: Front-end low-level architecture on Consensus Enterprises Blog published Thu, 06 Apr 2023 09:00:00 -0500 In our previous post, we talked about our recent client work building a Kubernetes-based system for hosting web applications. We’ve defined a general framework to support our development and production hosting workflows, and recognized this as a solid basis for an alternate backend to plug in to the existing Aegir5 front-end. Today we’ll take a look at the Drupal architecture underlying that front-end. In Aegir5, the building blocks consist of Task and Operation entities. Tasks represent … In our previous post, we talked about our recent client work building a Kubernetes-based system for hosting web applications. We’ve defined a general framework to support our development and production hosting workflows, and recognized this as a solid basis for an alternate backend to plug in to the existing Aegir5 front-end. Today we’ll take a look at the Drupal architecture underlying that front-end. In Aegir5, the building blocks consist of Task and Operation entities. Tasks represent relatively low-level reusable functionality. For example, we might have a Task for the Ansible back-end that is responsible for writing an Nginx vhost. We might then bundle that Task, along with others, in order to compose an “Install Drupal” Operation. Since the need to write a vhost is pretty common for web applications, that same Task might also be bundled into an “Install Joomla” Operation. Operations represent the actions that a user might want to trigger. Operations are configured to contain one or more Tasks. Each Task is a standalone fieldable entity. As a result each Task is responsible for gathering whatever data it needs to run its back-end tasks. The Operation includes these Tasks as inline entities. When we trigger an Operation, we are presented with a form that includes all the fields from the various Tasks associated with that Operation. These values, across all of the related Tasks, are marshalled and sent to the back-end. Logs are kept at the Operation level and so the output from the back-end gets piped back into a field on the Operation. In addition to the terminal output from the back-end, we also include the return code of the operation, so that we can record whether it succeeded or not. Another type of task is what we call an “action” task. These do not get bundled into Operations, but rather happen within a given page request. For example, instead of explicitly triggering a “password reset” task, as we do in Aegir 3, we can have a task dispatched when we click a “Log into site” button. This task runs a drush uli (via the Celery task queue), and relays the result directly back to the front-end, almost immediately. This can in turn redirect us directly to that URL. The result is that we are logged into the site without having to wait for a fully logged Operation to run. In the same way that Operations are made up of Tasks via inline entities, other entities such as Sites and Platforms are themselves made up of Operations embedded as inline entities. When we create a Platform entity, we instantiate all of its Operations and related Tasks. Among other things, this allows us to reuse Tasks between Operations. For example, when, taking a backup, we will want to record the file path to the tarball. This data then becomes available to the restore task that may come later. This case also illustrates the need for the back-end to be able to write data to the front-end. This is another function of relayd in the queue architecture. (see the upcoming post on this for more details). In addition to writing logs to Operation entities, it can write data to any field within any entity on the front-end. The sequence diagram below illustrates the flow from an Operation being triggered to run, through placing a task on the queue, running the backend executor (currently Ansible, soon Kubernetes), and getting feedback returned to the frontend. This architecture largely mirrors how Aegir 3 works. However, it provides a more flexible, robust and scalable solution. In our next post, we’ll take a look at the higher-level UI architecture we plan to implement to represent the key components in our Kubernetes hosting framework: Clusters, Projects, Releases, and Environments. https://consensus.enterprises/blog/aegir5-is-happening/ Sat, 25 Feb 2023 09:00:00 -0500 Aegir5 Development is Happening! on Consensus Enterprises Blog published Sat, 25 Feb 2023 09:00:00 -0500 Aegir5 development is happening! We (Consensus) have been making steady progress on it over the last few years and are looking to kick off a new burst of focused development. Here’s a summary of progress that has been made so far and how you can contribute. First off, as you’re probably aware, Aegir5 is a complete re-write of Aegir. We are intending to build on all the great aspects of Aegir, while freeing ourselves from a codebase that is rooted in PHP 4. We’re using D9+ alongside modern … Aegir5 development is happening! We (Consensus) have been making steady progress on it over the last few years and are looking to kick off a new burst of focused development. Here’s a summary of progress that has been made so far and how you can contribute. First off, as you’re probably aware, Aegir5 is a complete re-write of Aegir. We are intending to build on all the great aspects of Aegir, while freeing ourselves from a codebase that is rooted in PHP 4. We’re using D9+ alongside modern middleware and back-end components. The front-end (Drupal) and the queue system (based on Celery) are working fairly well. We’ve mostly reproduced the Aegir 3 behaviour, and it has a similar look and feel. This is the most recent work that’s been done directly in the Aegir5 repo, and dates back about a year, or more. (See for example aegir#94.) It’s suffering from some bitrot, so we’ll need to do a bit of cleanup. Bear in mind that we’re self-funding this project, so we aren’t currently able to dedicate much time and budget to it directly. However, we have had a number of contracts, both current and in the recent past, where we’ve been able to evolve some of the middleware and back-end components. For example, we recently worked with the Drupal Association to build Rugged, a TUF server implementation to secure the Drupal Composer repository. The Drupal Infrastructure team is in the process of beta testing this functionality on drupal.org, and is very much in line with the Aegir project’s goals of making updates easy and secure. More directly related to Aegir5, Rugged uses Celery extensively, and started out building from the components we’re using in Aegir5. We’ve developed them much further within the context of that project, and plan to port those improvements back to Aegir5. Likewise, our initial plan was to build the Aegir5 back-end using Ansible. While we have some basic playbook elements in Aegir5 currently, we’ve had a number of clients recently ask us to build fully automated Kubernetes-based hosting systems. Our first prototype is hosting a Django/React app, and is working well in production. We’ve been hosting our Kubernetes clusters on OpenStack, since we’ve been striving for a fully FLOSS stack as part of our overall goals for Aegir5. However, a number of our larger institutional clients are restricted to using one of the major cloud vendors. So we’ve been architecting our automation to support any Kubernetes cluster. We’re currently refining these components for a Drupal upgrade project for a department of the Government of Canada. The Drupal-on-Kubernetes piece was mostly completed a couple weeks ago. We’re continuing to enhance and stabilize it. Next, we’re planning to integrate something along those lines as the default back-end for Aegir5. We do not have a timeline for the integration of these components into Aegir5, mostly due to the aforementioned time/budget constraints. However, we are eager to proceed with that work. We have had a handful of organizations express interest in possibly funding some of this upcoming work, some of whom are currently hosting several hundred Drupal sites each on Aegir 3. If you know of someone who might be interested in helping to fund this work, please let us know. We’re undertaking to update the Aegir5 roadmap to reflect the progress we’ve made, and to outline next steps. Ideally we’ll be able to get a detailed enough break-down to be able to broadly estimate remaining effort for an alpha release. Hopefully this gives a good picture of the current state of Aegir5 development. We’ll be publishing some subsequent posts in the coming days to describe in more detail our Drupal-on-Kubernetes architecture, as well as how we aim to integrate the existing frontend and queue system work to achieve an MVP Aegir5. If you have any questions, don’t hesitate to ask. Aside from our Contact form, you can find us on LinkedIn or Matrix, and always eager to talk about Aegir! https://consensus.enterprises/blog/tuf_for_humans/ Tue, 10 May 2022 09:00:00 -0400 TUF for Humans: Explaining software update security on Consensus Enterprises Blog published Tue, 10 May 2022 09:00:00 -0400 For the past few months, we’ve been working with the Drupal Association on a project to enhance the security of the Drupal.org software repository. The most succinct way of describing this project is: Securing automated software deployments from supply chain attacks. Recently, on a long drive with my mother, I tried to explain this project to her. She is probably the least technical person I know. This may sound like the common tech trope, but it’s not. This has nothing to do with her age or … For the past few months, we’ve been working with the Drupal Association on a project to enhance the security of the Drupal.org software repository. The most succinct way of describing this project is: Securing automated software deployments from supply chain attacks. Recently, on a long drive with my mother, I tried to explain this project to her. She is probably the least technical person I know. This may sound like the common tech trope, but it’s not. This has nothing to do with her age or gender. She literally self-identifies as a neo-luddite. Over the course of that conversation, and subsequent ones, we came up with some helpful explanations and analogies that provide some insight into a very complex subject. So I figured that I’d share them here. Below, we’re going to cover the following topics: Software Supply Chains Validating Signatures Validating Documents Software Package Updates Verifying Packages Public-key Cryptography Digital Signatures Signing Packages First off, a new type of software security threat has been on the rise: supply chain attacks. In fact, by some estimates, they tripled in the last year alone. Some have been fairly high-profile, such as the recent Log4j and SolarWinds attacks. But what is a “supply chain attack”? Here the “supply chain” basically refers to all the software components that are used to build a modern application. Basically, instead of attacking a specific organization’s IT infrastructure, this kind of attack tries to inject malicious code into one of those software components. If it’s successful, then any organization that uses the compromized component is potentially vulnerable. One approach to address this type of threat systematically is The Update Framework (TUF). In the TUF Specification, the authors outline a host of attacks and weaknesses that they’re trying to address. They then go on to outline a rigourous process to secure software updates against these threats. The Update Framework is quite complicated. Even people with lots of experience with security, cryptography and package management often find it difficult to understand. Below, we try to explain some of the basic problems the TUF Spec is trying to solve, and how it suggests that we can address them. To help illustrate the nature of the problem space, let’s first consider some real-world examples: chequing accounts and contracts. When you go to a bank, to open a chequing account, they’ll generally ask you to sign a signature card that they’ll keep on file. When you then write a cheque, the bank will compare your signature on the cheque to the one they have on file for you. They should only transfer money out of your account if they match. The bank is verifying the authenticity of your signature. Another common place we find signatures is on contracts. Contracts are most often between two parties, and so have two signatures. On occasion, one of the parties might try to get out of their contractual obligations by claiming that they never signed the contract. If we were to point to their signature on our copy of the contract, they may claim it’s a forgery. The technical term for this is “repudiation”. How do we overcome this problem? One common approach is to have witnesses also sign the contract. Witnesses aren’t bound by the terms of the contract itself. Instead, their signature is meant to prove that the other parties did, in fact, sign the contract. If a contract dispute were to go to court, the witnesses may be called to testify to that effect. Each witness is asserting the authenticity of the signatures. It is also common to revise a contract’s terms. This often happens repeatedly before finally signing the contract. But it can also happen after a contract was already signed. So now we have multiple versions of a contract. What happens if the parties can’t agree on which version is authoritative? To avoid this problem, for really important contracts, a professional notary may be hired. The notary would review all copies of the most recent version of the contract and apply their seal and signature to each of them. The notary also keeps a logbook, where they register each time they notarize a document. These provide proof that all of the copies of the contract are identical. A notary’s seal is embossed into the paper of each copy of the contract. Only the notary has access to their seal. Older or altered versions of a contract would not have this seal. So they wouldn’t be considered valid. So the notary’s seal is validating the authenticity and integrity of the document (contract). These are simplified descriptions of these processes. But they should illustrate how signatures and seals can help to authenticate and verify a document. Let’s see how these principles can be applied to improving software security. Software security is, itself, a complicated subject. So we’ll start by describing a simplified example. Imagine that I have a simple blog website. Despite its outward simplicity, this website runs atop many, many components. We refer to this as a “software stack”. It’s a simplified version of the “software supply chain” that we looked at earlier. Such a blog website will likely be: Hosted on a server, probably running a Linux operating system; Served to your browser by a web server, such as Apache or Nginx; Written in a language like PHP or Python; Storing content in a database, such as MariaDB or Postgres; Built on a web framework like Drupal or Django; and Using various extensions and modules, to provide specific functionality. There are many undiscovered bugs and security flaws in the thousands of components that make up this stack. To keep my website secure, I need to promptly update any component that releases a new version that fixes a security flaw. Otherwise, if I leave the insecure code unpatched, an attacker might exploit it. For the sake of argument, let’s assume that my website was built using Drupal. Luckily, Drupal has a spectacular security team. They publish regular security advisories on a mailing list to which I’m subscribed. So I get alerted whenever I need to update my codebase. When I receive an alert that a module I’m using has a new security release, I’ll use a tool like Composer to download the latest version of the module and deploy the update to my website. After a couple more steps, my website has been updated, and the bug fixed. So far, so good… Except, attackers are getting increasingly sophisticated in how they go about compromising website security. What if, when I downloaded the new version of the module, someone managed to intercept the request, and instead delivered a hacked version of the module. This is a “Person-in-the-Middle” (PITM) attack. Now I’m actually worse-off than I was before, and I wouldn’t even know it. This is an example of what we defined earlier as a “software supply chain attack”. There are a number of similar types of attacks. For example, a “replay” attack is one in which, instead of downloading a hacked version of a module, I instead download an older version. This can be just as detrimental, and might be even harder to detect. But for the sake of simplicity, let’s just use PITM for our example here. What we need is a way to validate the authenticity, integrity and freshness of the packages that we’re downloading. This is where TUF comes in. Two components are required to verify package integrity: A TUF server, to sign packages (more on this below), and A TUF-enabled client, to verify the package signatures. Luckily, there’s a Composer plugin available to help with the second part. In our example, during the PITM attack, Composer can (via the TUF plugin) tell that the module has been tampered with. It can then display an error message, so that we’re aware of the problem, and react accordingly. The Composer TUF plugin basically steps in every time Composer downloads a file, and verifies the file’s integrity, freshness, etc. It does this by downloading information about the file, referred to as “TUF metadata”. This metadata includes information about the file, such as how big it should be. Most importantly, it contains a “hash” of the file. A hash is a long string of letters and numbers that uniquely identify a specific file. If you change even a single character in the file, then you’ll get a completely different hash. This hash acts like the notary’s seal. It allows us to easily verify that a given file has not been tampered with. But attackers are smart. How do we know they haven’t also altered the TUF metadata, to cover their tracks? This is where public-key cryptography and digital signatures come in. Here things start to get really tricky. Public-key cryptography relies on some very advanced math, and the way it works is quite unintuitive. As such, we’re only going to try to explain the most important element: asymmetry. Let’s start by looking at another real-world example: mail boxes. If you want to send me a letter, all you need is my address. However, for anyone to read that letter (once delivered), they’d need the key to my mailbox. Unlike my address, which is publicly available, I’m the only one with a key to my mailbox. This situation is asymmetric, meaning that it cannot be done in reverse. My mailbox key doesn’t help me to send a letter to you. Public-key cryptography works in a similar manner. It all starts by generating a “key pair”. A key pair consists of two files each containing a long string of characters (a cryptographic key). However, not just any two keys form a key pair. They need to be generated together, and have a very special relationship. If we encrypt a message with one of the keys, then it can only be decrypted by the other key in the pair. Notably, the key that was used to encrypt the message won’t help us to decrypt it. This holds true for both keys. Regardless of which one we use to encrypt a message, only the other one can be used to decrypt it. When we generate a keypair, we designate one of the keys as “public”, meaning it can be shared freely with others. We designate the other key “private”, meaning that it should never be shared at all. The public key acts like my address, in the example above. It allows anyone to encrypt a message that only I can decrypt, because I’m the only one with access to the private key. Digital signatures are basically the application of this same principle, but in reverse. Recall that either key can be used to encrypt a message that only the other key can decrypt. So, if I encrypt a message using my private key, then only the public key will be able to decrypt it. Remember that I’m the only person with access to my private key. So if I send you an encrypted message, and you can decrypt it with my public key, then you can be pretty sure that I’m the one that sent the message. Of course, this depends on me being responsible, and keeping my private key secure. We should now be familiar with all the concepts we’ll need to make sense of TUF. Earlier, we were concerned about whether an attacker might alter TUF metadata. Digital signatures are the solution. The TUF server treats the information (metadata) about the files we’re downloading as a message that it encrypts using its private key. Earlier, I left out that the Composer TUF plugin has a copy of the server’s public key. As a result, the plugin can decrypt the message, and thus be confident that it came from the TUF server. This means that the Composer TUF plugin can confirm the authenticity and integrity of the TUF metadata. Since it can trust the metadata, it can proceed to verify the files themselves. So a TUF server’s job is to: Generate metadata to allow a client to verify packages; Sign this metadata with private keys, so that the client can trust it; and Keep the private keys secure, so that attackers can’t forge metadata. Of course, this is all more complicated than what we’ve presented here. If you’re interested, I highly encourage you to read further on the TUF website As I mentioned at the very beginning, I’ve been working with the Drupal Association to build and deploy a TUF server to help secure downloads of modules and themes. This security infrastructure will become increasingly important as the Automatic Updates Initiative approches completion. One of the goals of this project has been to open-source the resulting code and documentation, to make it easier for other free/libre open source software (FLOSS) projects to adopt the TUF framework. We recently split out the TUF server component, that we’ve dubbed “Rugged”. Check it out at https://rugged.works