Scaling Puppet Environment Deployment

This the second post on a three part series on Puppet deployment.

In my original envisioning of dynamic environments with Puppet, I had a narrow vision to fit my current situation. It was simple enough - you would have one and only one repository, it would contain all of your manifests, and that would basically would be it.

As of the date of this writing there are 850+ Puppet modules on the Puppet Forge, and a few thousand modules on Github. On top of those raw counts the rate of module contribution is increasing and the quality of modules is steadily going up as people are figuring out how to make truly reusable modules. The adage goes “good coders code, great coders reuse,” so it makes sense to publish your good modules and reuse existing work.

So how do you roll existing modules into your deployment?

My team experimented with a number of implementations. Each attempt seemed to be the perfect fit and solve all our problems. It was only when we fully adopted one approach did we realize all of the crucial flaws with that approach.

Git Submodules

The original workflow was based on Git, so git submodules have been an obvious approach to the problem. Git submodules work by checking out a specific commit of a repository at a path within the containing repository.

I’ve already written a lengthy screed on git submodules, but the short of it is that git submodules have a number of shortcomings and they’re incredibly static. They’re an antipattern - they initially seem great but the more you start using git submodules, the more painful it gets.

Git Subtrees

When I wrote the blog post on git submodules I was using a new approach called git subtrees, which seemed to be the holy grail of merging git repos. The git subtree command actually wraps the “subtree” merge strategy. The git subtree merge works by taking two git trees, and shifting one to be contained by the other.

The good

Git subtrees have a number of advantages. First off, git subtrees merge in the entire history into the containing repository. Because of this, after you do the subtree merge it appears as if the merged repository was integrated into your repository the whole time. Manipulation of the subtree’d repository doesn’t require any special knowledge of git subtrees; you use git operations as normal and things just work.

The bad

Git subtrees have a pretty horrible flaw. The git subtree merge only shifts the contained repository once, at the time of the merge. The old history of the subtree’d repository is completely ignorant of the subtree merge.

One implication is that once you subtree a repository, you can’t rebase the history near that merge, ever. Git rebase works by resetting to a specific commit and then “replaying” the each commit change in the rebased history. This means that each git commit in the subtree’d repo is replayed at the root of the repository, not back into the directory where it was added. It’s really ugly, and there is no way of getting around it.

In addition, git subtrees inherit all the downsides as well as the upsides of merging two repositories. For instance, if someone committed binaries into a subtree’d repo, you’re now stuck with that binary blob in your git history for all time. If someone used sloppy commit messages, you have to live with that.


librarian-puppet is a project by the amazing Tim Sharpe to take Librarian, a general reimplementation of Bundler, and provide an implementation for the Puppet ecosystem. It has support for installing Puppet modules from the Puppet Forge as well as Github, and provides any number of other features like version locking of installed modules.

The good

Considering the tag line for librarian-puppet is “You can all stop using git submodules now,” it’s obviously a pointed approach at this problem space.

Librarian-puppet, like Bundler, really excels when it’s used in a somewhat limited scope. For instance, if you use librarian-puppet to install dependencies while developing on a module, you’ll be golden. Alternately if you’re deploying a set of modules that aren’t changing a lot, and only using that with a single environment, life is good.

The bad

However, we tried using librarian-puppet in a way that it was never meant to be used - deploying a large number of modules, many of which were frequently changing and all of which needed to be updated at once. Moreover we were using librarian-puppet to deploy multiple environments, and librarian-puppet start breaking down.

Librarian-puppet has a per-instance cache of git repositories, which it can use to speed up deployment time. However, the flaw here is per-instance; out of the box it has no concept of multiple environments sharing the same data. We were deploying about 70 modules out of git, all of which were generally shared across around 10 environments, and we couldn’t deduplicate that shared information. This meant that when we did a full update, we had to update around 700 git repositories, on each of 4 Puppet masters. I think Github wasn’t too happy about us periodically rebuilding 2800 git repositories, every 20 minutes or less.

In the end, librarian-puppet has a similar problem as git submodules themselves. Librarian-puppet implements the Bundler style lockfile which records the version information of every installed. Moreover, it will not deviate from the versions in the lockfile unless you update the lockfile. Even more moreover, if you delete the lockfile, librarian-puppet blows away ALL of the modules, and then rebuilds them, in parallel. My experience with this seemed to indicate that the librarian-cache wasn’t being fully used, so this would run a full reclone. Every time.

Librarian-puppet by itself is awesome, but if you misuse it the way I did, (shockingly!) it doesn’t work that well.

What now?

Clearly, everything is terrible and bad when it comes to deploying Puppet environments. So what do you do about this predicament?

Stand back, folks, I’ve got this.

Otherwise phrased, I got frustrated with this situation myself and wrote my own implementation for this environment. I’ll tell you all about it in the follow up blog post.