Home Approach Services Mission Team Careers Blog Contact

Drumkit and other plumbing

5 minute read Published: Author: Derek Laventure
Cloud , Infrastructure , Ansible , Drumkit

You are reading our blog series: Pure Ansible Infrastructure

Drumkit

Drumkit is one of my favourite tools that we use at Consensus, because it serves to unify and simplify how we interact with our project tooling. Drumkit itself is very simple, leveraging the venerable GNU make to create short, project-specific “targets” which are essentially a list of sub-commands to run. Any time we have a complex command-line command to do something, and we need to run it regularly, we add a Drumkit target for it and no longer have to remember the correct flags to consistently do the same thing.

In the case of our example Ansible infrastructure project, we leverage Drumkit for pinning a project-local installation of Ansible (and other tools) at a specific version, but primarily to provide easy access to key operations on the project itself:

  • make tools - ensure we have Ansible, as well as the Galaxy requirements in requirements.yml
  • make infra - run the playbook to create a Project, VPC and Firewall at DigitalOcean
  • make aegir - combine the 2 following targets to create and configure the Aegir droplet
    • make aegir-droplet - run the playbook to spin up a new droplet
    • make aegir-app - run the playbook to configure Aegir and its dependencies
  • make aegir-down - deregister the server from the VPN and tear down its resources

Installing Drumkit is easy enough, and using it is as simple as sourcing the bootstrap script in .mk/scripts/bootstrap.sh. Drumkit installs a symlink named d at the top of your project, so you can simply source d when you first move into the project, and Drumkit will set up your PATH to point to project-local tools, and run any scripts in drumkit/bootstrap.d/.

The Drumkit installer creates a number of files and directories you can commit into the repository:

  • .gitmodules & .mk - this is Drumkit itself, git submodule'd into the project. This is also where your project-local binaries go (.mk/.local/bin)
  • d - bootstrap.sh symlink
  • drumkit/bootstrap.d/*.sh - an initial set of simple boostrap shell scripts triggered by source d
  • Makefile - the top-level Makefile, which simply includes Drumkit’s .mk/GNUMakefile, that in turn includes the rest of Drumkit and your project’s make target definitions in drumkit/mk.d/*.mk.
  • drumkit/mk.d/ - the directory where your project-local Makefiles (.mk) will live. I typically create a 10_variables.mk in this file shortly after installing Drumkit to set the versions of tools required.

The bootstrap scripts checks for the environment variable DRUMKIT=1 to determine if Drumkit is already bootstrapped in this shell, and then sets that flag if so. It then adjusts your PATH to include the project-local .mk/.local/bin where any tools (like ansible) will be located after you run a make ansible or similar.

NB Git submodules are a powerful way to integrate different git repositories, but they do require that you subsequently clone your repository with the --recursive tree every time, in order to also clone the submodules into a new working tree. If you forget this step, you can do git submodule update --init --recursive to fix it :)

Python

Ansible is a Python-based tool, and so in addition to the basic Drumkit tooling, we need a basic Python environment within which to operate.

Let me say up front, I’m not a very skilled Pythonista. I speak a number of programming languages with varying degress of fluency, and I know enough that I would count Python among them. However, I’ve never been immersed enough in the community and culture of Python to really grok it. I am therefore conscious that I may be missing something entirely in the foregoing section, and hoping some savvy Python pixie will drop me a line to say “here, you can just do it like this” ;)

One thing I do understand about the modern Python is that it really wants you to do everything in a virtualenv. PEP 668: Marking Python base environments as externally managed is an example of this trend. For a Python-focused project using Drumkit, this presents a challenge with respect to bootstrapping. We need to activate a python virtualenv (essentially a sub-shell) and have Drumkit manipulate the shell environment. If we bootstrap Drumkit first, then its environment gets clobbered when we activate the virtualenv. However, I couldn’t find a way to “chain” the two together, so I could activate the virtualenv and have it immediately trigger a source d. Even better, I’d love to drop a drumkit/bootstrap.d/00_virtualenv.sh which would do the virtualenv setup as part of a regular source d.

What I’ve landed on is using pipenv for its simplicity and ability to manage dependencies and versions effectively. With a simple Pipfile in place to register the python library requirements Ansible will need, we can get started in the project like so:

pipenv shell # Activate virtualenv
. d          # Bootstrap Drumkit
make tools   # Install ansible and galaxy requirements

Note the source d includes a pipenv install from drumkit/bootstrap.d/10_pipenv.sh to actually install the Pipfile requirements into the virtualenv.

Plumbing

As a developer, streamlining my workflows and making my daily tools quick and comfortable to use is essential to have the “plumbing” of my project just get out of the way, and let me focus on the problem at hand.

With this Drumkit and Python virtualenv setup in place, I have a shared set of versions and tailored-commands for common operations in the project, and I can extend it readily.

Having done the bootstrapping steps above, I can run any Ansible tool like ansible-galaxy or ansible-inventory, or equally any of the currently-available Drumkit targets. As I build out more pieces of the project, I implement new Drumkit targets in simple modules under drumkit/mk.d to provide easy “wrappers” around commands. Thus, instead of remembering to type ansible-playbook playbooks/hosts/aegir0.yml && ansible-playbook playbooks/apps/aegir.yml, I can simply run make aegir.

We now have the foundation of our Ansible project in place. In the next entry in this series, we’ll build on this tooling to implement Ansible dynamic inventory.

The article Drumkit and other plumbing first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!