How Dynamic Environments Came to Be

This is the first post in a three part series on Puppet deployment.

My first interaction with Puppet was when I was a junior sysadmin at my University. One of the previous lead Unix sysadmins had dabbled a little with Puppet when Puppet itself was a very new tool, and his work spurred the use of configuration management at the university. More and more of the infrastructure became managed with Puppet and it became a fundamental part of day to day operations and was critical to the smooth functioning of the Unix team.

It’s the kind of story you tend to hear all over the place when asking how people came to use Puppet.

Access to the Puppet manifests was fairly tightly controlled. After all, if you had access to the Puppet manifests running on a machine then you basically had root on the machine, so it made sense to lock things down. People had to show up and show some merit in the organization before they were given read access to the git repository containing the manifests. They were then encouraged to learn about Puppet and make contributions, but a senior sysadmin had to review and merge your code.

The development workflow for new sysadmins looked something like this. You would clone the git repository, create a new branch, and try to figure out where to start. You would make changes based on your limited understanding of Puppet, commit it, and then track down a senior sysadmin to review and merge your code. If bugs or flaws were found you would start from ground zero. Eventually your code would pass inspection and get merged - and if you were unlucky then it would fail in production and get reverted, and you were back to square one.

Another reason access to the git repository was so tightly controlled was that code deployment was triggered via a post-receive hook on that repository - when you pushed to that repository, the code went live. This was very convenient for quick iteration, but it made push access very sensitive.

In this story I played the part of the brash young sysadmin, chomping at the bit to get push access to the Puppet manifests. I wanted to work independently and test things on my own without having to pester other people or have my hand held while working - I just wanted to get things done. I had recently worked on a Ruby on Rails project and by default RoR uses three environments: dev, test, and prod. My idea was to replicate this structure using three git repos with three post-receive hooks. I could push to dev and test, and production was still secure. After much nagging of senior sysadmins the idea was implemented and I was partially set loose on the infrastructure. Life was good.

Well, life was good until two different tests were made to the same environment. We only had three environments, so while I was hacking on a feature and pushing to the development environment, perhaps someone else was working an another feature and also pushing changes to the development environment. If a mistake was made with one feature, everybody was affected. Even worse, if someone merged their changes in development into production, they may have just merged in incomplete work. It was not ideal.

Git has these problems completely nailed down. Git makes very heavy use of branches to stabilize the main branch, isolate change to discrete sets of commit, facilitate reviews, and so forth. I had used Git in this manner quite a bit in the past, and I loved the experience. And yet here we were, using Git with Puppet as if it was SVN with a set number of branches.

At some point some of my coworkers and I got to talking - wouldn’t it be nice if we could adapt the Git workflow and make environments on the fly with Puppet? It would solve all our problems! But how would this work?

As it turns out, implementing dynamic environments in Puppet is surprisingly easy. The Puppet config file supports variables in strings, so you can interpolate values in as needed. It’s good for short hand variable assignments like this:

vardir = /var/lib/puppet
ssldir = $vardir/ssl

In static environments, you would have a set list of environments like this:

  server =
  environment = production
  confdir = /etc/puppet
  manifest = /etc/puppet/environments/production/manifests/site.pp
  modulepath = /etc/puppet/environments/production/modules
  manifest = /etc/puppet/environments/testing/manifests/site.pp
  modulepath = /etc/puppet/environments/testing/modules
  manifest = /etc/puppet/environments/development/manifests/site.pp
  modulepath = /etc/puppet/environments/development/modules

The critical thing here is that the environment is also exposed as a variable. So instead of having a static list of environments, you could do this:

  server =
  environment = production
  environment = production
  manifest    = $confdir/environments/$environment/manifests/site.pp
  modulepath  = $confdir/environments/$environment/modules

After we discussed how this could work, the lead sysadmin decided that it was worth a shot, updated the puppet.conf accordingly, and created a basic post-receive hook. When the whole thing was turned on, it just started working, and it worked flawlessly. It caused a fundamental shift in development - it was much easier to try out things, code was generally higher quality, it was just plusses all around.

One of the former sysadmins who got a job at Puppet Labs found out about our approach, and he liked it so much that he started telling Puppet users about it. Everywhere it went it was a hit. (He actually wrote a blog about it, but as of the time of this writing it seems to be down.) Amusingly enough it came full circle - another Puppet Labs employee wound up coming around the university and started showing people this incredibly cool thing he had heard about. We had to gently break the news to him that it was invented here.

Later on, I wound up joining Puppet Labs myself, and I got the chance to implement this workflow inside of Puppet Labs operations. Again it was a hit, and I wound up writing a post on the company blog to help spread the word, and to this day it’s one of the more popular posts on that blog.

In fact, it was writing that post that got me interested in creating this blog.

The dynamic environments workflow was a killer idea, and it worked well for over two years for me. However as we tried doing more complex things with deployments, this approach ran into scalability issues. In my next blog post I’ll explain those scaling issues, and in the following blog post I’ll talk about the new tool that I’ve written to solve the Puppet deployment problem.