Throughout my internship at CustomInk, I've put a significant focus on Chef cookbook testing. At the time of this writing, there are a few solutions for testing cookbooks - ChefSpec, cucumber-chef, minitest-chef-handler, and rspec-chef – and they each have their own distinct advantages. At the very least, you should run knife cookbook test and foodcritic against all your cookbooks. Nathen Harvey covered this in his MVT: knife test and TravisCI blog post.
At CustomInk, we test using ChefSpec. Additionally, we use some home-grown gems such as fauxhai and Strainer to make testing easier.
Foodcritic is a linting tool for your cookbooks. Although technically not a "test", linting tools are frequently grouped with testing. Foodcritic is like jslint for cookbooks. At the bare minimum, you should run foodcritic against all your cookbooks.
As mentioned in Nathen Harvey's MVT: knife test and TravisCI, foodcritic does not verify that you have proper Ruby code. It is only a linting tool.
When running foodcritic, I recommend adding both CustomInk foodcritic rules and Etsy foodcritic rules. Clone the repositories (or use submodules) into a foodcritic directory in the root of your chef-repo:
|_cookbooks
|_data_bags
|_environments
|_foodcritic
|_customink
|_etsy
Now, you can run foodcritic like so:
$ bundle exec foodcritic -I foodcritic/* cookbooks/my_cookbook
There are more details about the CustomInk foodcritic rules and Etsy foodcritic rules in their respective repositories on github.
Very quickly, you can see the need to run multiple commands against every cookbook:
knife cookbook test for ruby syntaxfoodcritic to lint your codechefspec, minitest, or cucumber for your actual unit, function, or integration tests
$ bundle exec knife cookbook test COOKBOOK $ bundle exec foodcritic -I foodcritic/* cookbooks/COOKBOOK $ bundle exec spec
This is why I wrote Strainer. Strainer uses a Colanderfile at either the project-level or cookbook-level to run isolated tests on your cookbooks. The cookbook is actually copied to a temporary location and then the tests are run against it. To get started, create a Colander file in your editor and enter the following:
knife test: bundle exec knife cookbook test $COOKBOOK
foodcritic: bundle exec foodcritic -I foodcritic/* cookbooks/$COOKBOOK
chefspec: bundle exec spec
Notice that we have this $COOKBOOK variable. There's also a $ENVIRONMENT variable provided. These are replaced with the cookbook or cookbooks you are straining.
If you've used foreman, you should be familiar with this syntax.
Next, we want to strain out the cookbook we want to test:
$ bundle exec strain my_cookbook
Strainer is also useful for setting up multiple builds or jobs against your single chef repository. You can create a separate job (on Jenkins for example), with the same base repository, but change the script (or use environment variables) to only run the cookbooks you want to test!
I recommend also checking out Strainer on github for the most up-to-date documentation. There's still a lot of work to be done. Get the bottom of the README for the most up-to-date list of features that need your help!
Problem: Ohai is awesome. Ohai does a LOT. I don't want to type out all my ohai data for a single test.
Solution: Fauxhai
Fauxhai is "fake ohai", as it's name implies. Fauxhai, as featured on the foodfight show multiple times is a community-backed node-mocking utility. In short, it looks like this.
Fauxhai.mock(platform: 'ubuntu')
This will mock out the default ubuntu node. You can use this in combination with Chefspec or other testing tools.
I recently released the first non-alpha version of fauxhai. In this edition, fauxhai is actually capable of mocking "real" nodes. It will SSH into your server, download the real Ohai data, and then run your tests against that Ohai data:
Fauxhai.fetch(url:'node.example.com')
Check out the fauxhai github page or fauxhai github repo for more information on how to use fauxhai.
At CustomInk, we test our cookbooks with ChefSpec by Andrew Crump. There are a lot of great testing solutions out there, but ChefSpec is what works for us and our needs as a company. We also rarely use ChefSpec without fauxhai, so here's an example from our phantomjs cookbook:
require 'chefspec'
require 'fauxhai'
describe 'phantomjs::default' do
context 'on ubuntu' do
before{ Fauxhai.mock(platform:'ubuntu') }
let(:chef_run){ ChefSpec::ChefRunner.new.converge('phantomjs::default') }
it 'should install the correct packages' do
chef_run.should install_package 'fontconfig'
chef_run.should install_package 'libfreetype6'
end
end
end
It's simple, it's explicit, and it reads a whole lot easier than TestUnit or MiniTest (shhh, don't tell DHH).
There is one hang-up about ChefSpec - it has incredibly confusing documentation. Depending on where you look, some methods have changed, syntax is different, things have been removed, and there's a few different places for documentation. The solution - just dive into the source. It's the easiest way, and you'll have a better understanding from the start.
Internally, we use Jenkins as our build server. However, for all our public-facing cookbooks, we use Travis. This causes somewhat of a gap between our build statuses - we have to look in two locations to see the status of all our builds...
Okay, I lied. We don't. We actually use another project I wrote, stoplight, that shows the build status from multiple build servers in a single UI. It's incredibly handy for these kinds of situations. And, for you hardcore Chef users, there's even a Chef cookbook for stoplight for installing stoplight on your infrastructure.
Chef testing is a huge topic, and it's going to be awhile until the community agrees on a single method for testing. Rails still has yet to figure it out... Here's what is important - test your cookbooks. Even if it's just running foodcritic against all your cookbooks, you are making the cookbook-world a better place. And talk about it. Tutorials like this one, as well as some of the other posts I've written will help the community embrace testing :)!