Alternate title: How I Learned to Stop Worrying and Love Cthulhu. Ia! Ia! Cthulhu fhtagn!

Every time I dive into the source code of Puppet, I vacillate between rapt awe at the sheer brilliance and power of the code, or I start losing my sanity. There’s so much cool behavior built into the entire system that allows you to do incredibly cool things. There’s also incredibly old code (see TransObject and TransBucket) that’s a little frightening. There’s also code so complex, that my face melts right off, Indiana Jones style.

So why am I doing this? Why am I diving into this valley of madness?

If you’re writing a simple type and provider, you can probably use existing types and providers on which you base your code. If you’re just executing commands, you’ll probably be in good shape. If you’re manipulating files that are record based you can probably use the ParsedFile provider. So we’ve optimized a lot of work for the simple cases, but when you want to do something more advanced, suddenly people start wondering when you’ll be checked into a mental institution.

I present the case where you’re not dealing with simple records; you have a fairly complex structured file. The ParsedFile provider doesn’t do so well with that - it tries to handle file manipulation and content parsing in one fell swoop. The file manipulation stuff is pretty cool - it supports prefetching, instances, and has a single pass flush method, so updating N resources in a file only takes one write. However, the FileParsing code that implements the DSL stuff for parsing gets very complicated.

I’ve been working on extracting the file manipulation logic and completely avoiding the parsing bit with my filemapper provider mixin. The idea is that you have a mixin do everything needed to manage the resources - that is, do the actual reading and writing, instantiate resources, et cetera. It shouldn’t ever be aware of the actual content being handled and should act in the most general way fashion. Classes that include this take over the burden of parsing and then generating files entirely.

Now the fun bit is that the ParsedFile provider uses methods and hooks that aren’t really documented or explained. You you might see something like mk_resource_methods and then descend into madness tracing this down.

There’s also shit like this:

def destroy
  # We use the method here so it marks the target as modified.
  self.ensure = :absent
  (@resource.class.name.to_s + "_deleted").intern
end

Guys, I don’t have the FAINTEST FUCKING IDEA why that string is being returned. I don’t even know if the return value for destroy is used at all, in which case the last line is meaningless.

This highlights the major problem; there is documentation on how to write new providers, but there’s very little documentation of the API. For attribute accessors, the contained value is not documented and can’t be easily derived; for methods, the arguments or returned values are not documented; while there might be a brief description of the purpose of a method, it’s not guaranteed.

This isn’t the fault of anyone, really. Puppet’s a very big, powerful tool, and our docs guys crank out tons of fantastic docs. Our engineering team also does amazing work, and they are why Puppet kicks so much ass.

But the issue is that there’s a little bit of a bisection between the engineers that work on core, and the people that work on types and providers. The core guys know the entire inside and out and they’re not necessarily focusing on the interaction between types and providers. And then you have someone like me, that just wants to know how a property from a type gets applied, and you get at least one blog post and a number of hours burned digging through source code.

As I’m going through all of this code, I’m hoping to start documenting the classes I go through here and explain what’s going on.