Reading Puppet: Pluginsync

In my previous post on the Configurer, I only made a passing reference to pluginsync. However, pluginsync is rather important both because it’s the mechanism in which Puppet can be extended, but it’s a really interesting window into how Puppet configures itself. We’ll be looking at Puppet::Configurer::PluginHandler and Puppet::Configurer::Downloader in this post.

PluginHandler is a module that contains the basic pluginsync logic, and it’s mixed in to the Configurer. It’s a convenient way to extract that logic out of the Configurer. The really interesting bit is the Downloader, because it shows how Puppet can use the same mechanisms to configure your system to configure itself.

Puppet::Configurer::PluginHandler

The PluginHandler is actually rather small; only implementing the following methods:

To kick off things, in Puppet::Configurer#prepare, the #download_plugins method is called.

Puppet::Configurer::PluginHandler#download_plugins

The logic behind this method is pretty simple. It calls the #download_plugins? method to see if pluginsync is enabled. If /etc/puppet/puppet.conf has pluginsync = true set, or it’s being run with –pluginsync true then it’ll continue with the sync. If pluginsync is enabled, it instantiates a new Puppet::Configurer::Downloader object, evaluates it, and for each plugin synced, calls the #load_plugin method. Straight forward, right? (Oh yeah, I’m totally skipping over the #evaluate call. We’ll get there shortly.)

Puppet::Configurer::PluginHandler#load_plugin

This method does some basic sanity checking around loading plugins. It makes sure that the plugin is actually loadable, and then tries to load it. If you’ve been developing on a type or provider and made a syntax error, this is where it’ll explode. However, if the actual load fails, the method will capture that failure, throw it away, and continue with the rest of the pluginsync and run.

So how exactly do plugins get transferred from the master to the agent? The answer is, MAGIC! Oh, and the Downloader helps too. But it’s magic.

Puppet::Configurer::Downloader

So this is where things get interesting - Puppet actually creates a new catalog to perform pluginsync. While there’s a lot of talk around Puppet compiling a manifests into a catalog and that’s what’s applied to your system, catalogs can be used for more than just that.

When you think about it, what is pluginsync? Really, it’s just a recursive file definition for your libdir. That is, it looks like this:

file { $libdir:
  ensure  => directory,
  recurse => true,
  source  => 'puppet:///plugins',
  force   => true,
  purge   => true,
  backup  => false,
  noop    => false,
}

Heh, so funny story - that’s exactly what’s happening. When you get down to it, the Downloader is just a thin layer around a very simple catalog that contains this one resource. From here on out, things act like you would expect.

So the PluginHandler creates a Downloader, and then calls #evaluate on it. What’s happening?

Puppet::Configurer::Downloader#initialize

When the PluginHandler creates a new Downloader, it sets up some basic instance variables. They are as follows:

You’ll notice that all of these are all centered around parameters for a File resource. That’s exactly what’s going on.

Puppet::Configurer::Downloader#file

This method generates a single File resource, and passes the previously mentioned variables to it. It sets up some specific things and does some variable munging, but the core of it is this:

Puppet::Type.type(:file).new(args)

This takes the variables specified in #initialize and generates the File resource.

Puppet::Configurer::Downloader#catalog

So we have this simple file resource, now we need to add it to a catalog so we can do something with it. This method does just that - but with some exceptions.

ENTERING TANGENT LAND

With a normal catalog, called a “host catalog”, special processing is done. Reports are generated and sent, graphs are made, and the normal operations are done. However, this is just internal Puppet configuration, so you probably don’t care about a report of just this. This catalog is what is known as a “resource catalog” and it’s treated differently inside of Puppet.

This is how Puppet can configure itself, and still retain some semblance of sanity - host catalogs are what you apply to a host, and resource catalogs are what Puppet applies to itself.

EXITING TANGENT LAND

Anywho, this will spit back a resource catalog containing the single file resource created above.

Puppet::Configurer::Downloader#evaluate

When the PluginHandler created a Downloader instance, it immediately called #evaluate. This is the meat of the Downloader; it creates the catalog and applies it. The libdir will be recursively synced, and the list of files that were changed during this process will be recorded and returned.

This recording of the files that were changed is how Puppet only loads files that were changed during pluginsync. You’ll notice that when you run Puppet the first time with pluginsync, a number of files will be loaded following the sync. However, successive runs won’t display this. It only makes sense to reload files that were changed, so this is how it’s implemented.

If you use the dynamic puppet environments from git branches workflow, you may have noticed something odd if you’ve ever tried to run against an environment that doesn’t exist. The way the workflow works is that any environment name works, but only environments that exist will actually do something sane. If you try to run Puppet against an environment that doesn’t exist, you’ll get this:

root@modi:~# puppet agent -t --environment NOTANENVIRONMENT
info: Retrieving plugin
err: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve information from environment NOTANENVIRONMENT source(s) puppet://puppet/plugins
notice: /File[/var/lib/puppet/lib/puppet]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/provider]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/database_user]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/provider/database_user]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/database_user/mysql.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/provider/database_user/mysql.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/network_config]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/provider/network_config]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/java_ks]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/provider/java_ks]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/modules]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/modules]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/vc_folder]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/provider/vc_folder]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/file_line]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/provider/file_line]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/test]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/test]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/firewall.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/provider/firewall.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/sysctl]: Dependency File[/var/lib/puppet/lib] has failures: true
warning: /File[/var/lib/puppet/lib/puppet/provider/sysctl]: Skipping because of failed dependencies
[etc]

From my understanding, this is the effect of this bug. Effectively, the environment and subsequent modulepath is valid but there’s no plugins to be synced, since the directories are unavailable. Since there’s no directory, every single plugin that’s been synced explodes. If you have puppetlabs-stdlib or a number of custom types/providers/functions, this will be a very lengthy explosion. My hat is off to Patrick for resolving this bug and others; he’s done a fantastic job of improving code around Puppet and environments.

So that’s it! At the end of the day, pluginsync is just a File resource using a very simple catalog. These are mechanisms you’ve certainly been using all over the place, but put to use in a really clever way.

Addendum: Puppet 2.7.17 was used as the reference version for this blog.

« previous
blog ↑ listing
next »