Berkshelf and Chef Cookbook Dependencies

I continue delving into the specifics of management automation and configuration, while also trying to share my community experience. In this article, I will continue talking about the automation tool for solving Chef cookbook dependencies, namely Berkshelf.

What does Berkshelf have to do with this?

Chef has a significantly large and actively developing community that constantly contributes to creating and updating cookbooks. They are all stored on the community website, and we often use many of them.

But, there is a catch in our company regarding editing community cookbooks. The correct way of applying any changes related to the specifics of the corporate infrastructure is creating wrappers with updates (for example, reassigned attributes, switched recipes and so on). In brief, below is my description of how to create a wrapper.

However, the fact that a correct path exists yet does not mean that everyone will follow it. For this reason, at some point our corporate cookbook repository gathered a large number of non-conventionally edited community cookbooks with misplaced edits that should be rather located in the wrapper than in cookbooks. So the time has come to clean up the repository.

The plan was as follows:

  • Select community cookbooks in our repository.
  • Identify the quantity of local updates (use diff to find the differences between clean community cookbooks and the spoiled versions).
  • Collect the updates in the wrapper and add a request into it to call the community cookbook or one of its recipes.
  • Remove the community cookbook and add it to the wrapper's dependencies.

But what do we do with the "cleaned up" community cookbooks? After the removal, how do the cleaned up cookbooks appear on the Chef Server?

This is where Berkshelf can help.

What is Berkshelf and how does it work?

Berkshelf is a dependency manager for Chef cookbooks. It is written in Ruby and has the methods and the API for interacting with the Chef Server. Berkshelf is installed either from Ruby Gems or using Chef DK. We use Berkshelf 2.0, but the recently released version 3.0 introduced several changes and "goodies".

If you choose Ruby Gem, I suggest installing it within the framework of Ruby installed by the Chef Server (executable files are located here: /opt/chef/embedded/bin/). You must configure Berkshelf for it to be able to interact with the Chef Server, define the address of our server and the authorization certificates. To configure Berkshelf, run the following command:

berks configure

Configuration files are stored in one of the following locations:

- $PWD/.berkshelf/config.json
- $PWD/berkshelf/config.json
- $PWD/berkshelf-config.json
- $PWD/config.json
- ~/.berkshelf/config.json

After Berkshelf is correctly installed, you can move on to resolving cookbook dependencies. For this purpose, Berkshelf first copies the cookbooks and their dependencies to its shelves (it is a local Berkshelf storage/repository, by default it is the ~/.berkshelf/cookbooks/ directory), and then loads them to the Chef Server.

At this point, you are probably wondering about many things: "How does Berkshelf know about the dependencies? What does it do to resolve them?"

All instructions for Berkshelf are stored in the Berksfile located in the cookbook's root directory. Create this file by running the following command within the root directory:

berks init

The contents of this file describe the dependencies and the source for resolving them. An example of a Berksfile:site :opscode

metadata cookbook 'my-cookbook', (:path | :git | :github) cookbook 'my-book-2', ('> 1.0.0')

This file is to do the following:

  • Recognize the dependencies from the metadata.rb file of our cookbook and load them from the Opscode Community website.
  • Load the my-cookbook folder from the location defined in brackets (it can be a local path or a link to Git).
  • Load the latest version (higher than 1.0.0) of the my-book-2 file from the Opscode Community website.

Afterwards, you can run the following command:

berks install

and wait for the dependencies to be successfully copied to the Berkshelf file's local storage. As a result, the storage directory should contain all cookbooks mentioned in the depends field of the metadata.rb file, their dependencies (the dependencies' dependencies) and two cookbooks mentioned in the Berksfile. Verify the result with the following command:

berks shelf list

Next, run the following command:

  • berks upload

It will load everything from the local storage to the Chef Server. As a result (given that the Chef Server is available and we can authorize it for using the certificate files), all the dependencies should be resolved. Verify the result of the upload with the following command:

knife cookbook list

Its output should contain new cookbooks.

Essentially, this is the basic process of using Berkshelf. Of course, that's not the entire functionality, as we don't touch the interaction between Berkshelf and Vagrant, Chef Solo, Chef Client, as well as some of the improvements introduced in version 3.0. However, I consider this amount of information to be sufficient for the majority of users.

Our experience with Berkshelf

It all would be totally great if not for one thing - Berkshelf cannot work cyclically. I was very surprised, because I was unable to find an option that would make it "natively" consider the nested behavior of directories and cookbooks. What am I talking about? For example, imagine the most typical situation where you have the chef-repo directory, and the cookbooks directory with our cookbooks (./chef-repo/cookbooks/) is nested into it. In the current version of Berkshelf, you must go to each of the cookbook's directories to run commands aimed at Berkshelf. Namely - write the bash script that would do it manually, seriously? It is a solution, but not the best one, to say the least – I would go as far as calling it a "crutch".

On the Internet, I found an article about how to solve this issue using Ruby.

The following is the code of our "root" Berksfile stored in the ./chef-repo/ directory:

site :opscode
def dependencies(path)
berks = "#{path}/Berksfile"
instance_eval( if File.exists?(berks)

Dir.glob('./cookbooks/*').each do |path|
dependencies path
cookbook File.basename(path), :path => path
end cookbook 'gecode', '= 2.1.0'