Throughout my internship at CustomInk, I’ve put a significant focus on Chef cookbook testing. At the time of this writing, there are a few solutions for testing cookbooks - ChefSpec, cucumber-chef, minitest-chef-handler, and rspec-chef – and they each have their own distinct advantages. At the very least, you should run
knife cookbook test and
foodcritic against all your cookbooks. Nathen Harvey covered this in his MVT: knife test and TravisCI blog post.
Foodcritic is a linting tool for your cookbooks. Although technically not a “test”, linting tools are frequently grouped with testing. Foodcritic is like jslint for cookbooks. At the bare minimum, you should run
foodcritic against all your cookbooks.
When running foodcritic, I recommend adding both CustomInk foodcritic rules and Etsy foodcritic rules. Clone the repositories (or use submodules) into a
foodcritic directory in the root of your chef-repo:
|_cookbooks |_data_bags |_environments |_foodcritic |_customink |_etsy
Now, you can run
foodcritic like so:
$ bundle exec foodcritic -I foodcritic/* cookbooks/my_cookbook
Very quickly, you can see the need to run multiple commands against every cookbook:
knife cookbook testfor ruby syntax
foodcriticto lint your code
cucumberfor your actual unit, function, or integration tests
$ bundle exec knife cookbook test COOKBOOK $ bundle exec foodcritic -I foodcritic/* cookbooks/COOKBOOK $ bundle exec spec
This is why I wrote Strainer. Strainer uses a
Colanderfile at either the project-level or cookbook-level to run isolated tests on your cookbooks. The cookbook is actually copied to a temporary location and then the tests are run against it. To get started, create a Colander file in your editor and enter the following:
1 2 3
Notice that we have this
$COOKBOOK variable. There’s also a
$ENVIRONMENT variable provided. These are replaced with the cookbook or cookbooks you are straining.
If you’ve used foreman, you should be familiar with this syntax.
Next, we want to
strain out the cookbook we want to test:
$ bundle exec strain my_cookbook
Strainer is also useful for setting up multiple builds or jobs against your single chef repository. You can create a separate job (on Jenkins for example), with the same base repository, but change the script (or use environment variables) to only run the cookbooks you want to test!
I recommend also checking out Strainer on github for the most up-to-date documentation. There’s still a lot of work to be done. Get the bottom of the README for the most up-to-date list of features that need your help!
Problem: Ohai is awesome. Ohai does a LOT. I don’t want to type out all my ohai data for a single test.
This will mock out the default ubuntu node. You can use this in combination with Chefspec or other testing tools.
I recently released the first non-alpha version of fauxhai. In this edition, fauxhai is actually capable of mocking “real” nodes. It will SSH into your server, download the real Ohai data, and then run your tests against that Ohai data:
At CustomInk, we test our cookbooks with ChefSpec by Andrew Crump. There are a lot of great testing solutions out there, but ChefSpec is what works for us and our needs as a company. We also rarely use ChefSpec without fauxhai, so here’s an example from our phantomjs cookbook:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
It’s simple, it’s explicit, and it reads a whole lot easier than TestUnit or MiniTest (shhh, don’t tell DHH).
There is one hang-up about ChefSpec - it has incredibly confusing documentation. Depending on where you look, some methods have changed, syntax is different, things have been removed, and there’s a few different places for documentation. The solution - just dive into the source. It’s the easiest way, and you’ll have a better understanding from the start.
Travis & Jenkins
Internally, we use Jenkins as our build server. However, for all our public-facing cookbooks, we use Travis. This causes somewhat of a gap between our build statuses - we have to look in two locations to see the status of all our builds…
Okay, I lied. We don’t. We actually use another project I wrote, stoplight, that shows the build status from multiple build servers in a single UI. It’s incredibly handy for these kinds of situations. And, for you hardcore Chef users, there’s even a Chef cookbook for stoplight for installing stoplight on your infrastructure.
The Long & Short of it
Chef testing is a huge topic, and it’s going to be awhile until the community agrees on a single method for testing. Rails still has yet to figure it out… Here’s what is important - test your cookbooks. Even if it’s just running foodcritic against all your cookbooks, you are making the cookbook-world a better place. And talk about it. Tutorials like this one, as well as some of the other posts I’ve written will help the community embrace testing :)!