The best - and sometimes the worst - thing about Chef is its thriving ecosystem. There appear to be cookbooks for just about everything, including one just for installing emacs. And there are lots of tools, lots of instructions, and blog posts about using the tools. So many in fact that it is sometimes hard to sort out what is what - especially as many of them are focused on showing you an example project rather than talking about broad concepts. I have picked up a number of useful tidbits but it’s been a little tough to integrate everything.

So, let me start with what I think I want. I want to use chef to set up my Linode VPS. Before I make changes to the server, I would like to run the same recipes locally on a Vagrant VM and run some integration tests to check that I am going to get what I intend. Eventually I may want to add unit tests for my custom cookbooks but for now, I want to just focus on ‘outside in’ testing which, in this situation, seems to me to be integration tests. This blog post, Getting Started Writing Chef Cookbooks the Berkshelf Way, Part 3, appears to do what I want to do, so let’s give that a try.

Initialize

I was originally going to manage everything using chef solo because I thought that would be easier. But it turns out that chef and all the tooling around it are optimized for using chef server to manage lots of nodes, so chef server is actually the easier option - especially since Opscode has a free tier for managing up to five nodes.

When I created my Linode VM with knife-linode I did it all on the command line - no files. So that means that while I have a node named ‘linode’ on my hosted chef and my Linode VPS is set up as a client of that node, I don’t have any files and no git repository keeping track of anything. So let’s see if I can construct something to connect to that node on https://manage.opscode.com/. So I stared with the chef gemset I had previously set up and upgraded (chef 11.10.0, kitchen-vagrant 0.14.0, test-kitchen 1.1.1).

    $ mkdir ~/chef-linode
    $ cd chef-linode
    $ kitchen init
          create  .kitchen.yml
          create  test/integration/default
             run  gem install kitchen-vagrant from "."
    Successfully installed kitchen-vagrant-0.14.0
    Parsing documentation for kitchen-vagrant-0.14.0
    Installing ri documentation for kitchen-vagrant-0.14.0
    1 gem installed

kitchen init created the .kitchen.yml file which is used to configure my test kithcet setup. I added the port forwarding I will need once I have a web server running on my VM. It is configured to spin up a Vagrant VM with the Opscode Ubuntu 12.04 provisionless VM and then use chef_omnibus to install the latest version of chef and provision with chef_solo. I am not sure whether I should be using chef_solo (so I can test out my cookbooks before they get uploaded to Opscode) or if I should change that configuration to point to the linode node on my chef server.

First server spec

So based on the example I worked through from the TDD book, and some other blog posts, I created a spec file in chef-linode/test/integration/default/serverspec/emacs_spec.rb

    require 'spec_helper'

    describe "emacs cookbook" do
      it "should have installed emacs" do
        expect(package 'emacs23-nox').to be_installed
      end
    end

I think this should indicate to test kitchen that I want it to use serverspec to check that I have installed emacs on the VM. Right now, I don’t have a spec_helper file; according to the Getting Started section, I get one by doing serverspec-init. But I am a little unclear about where to put it. The serverspec docs indicate I should have the directory a directory structure like:

    |-- spec
    |   |-- spec_helper.rb
    |   |-- <servername> # e.g localhost or cynthiakiser.com
    |       |-- emacs_spec.rb

But the blog post I am following, shows a directory structure of:

    |-- serverspec
    |   |-- spec_helper.rb
    |   |-- <servername> # e.g localhost or cynthiakiser.com
    |       |-- emacs_spec.rb

And I need a spec_helper.rb. serverspec-init will generate one:

    (brazen:~/chef-linode/test/integration/default) $ serverspec-init
    Select OS type:

      1) UN*X
      2) Windows

    Select number: 1

    Select a backend type:

      1) SSH
      2) Exec (local)

    Select number: 2

     + spec/
     + spec/localhost/
     + spec/localhost/httpd_spec.rb
     + spec/spec_helper.rb
     + Rakefile


    (brazen:~/chef-linode/test/integration/default) $ cat serverspec/spec_helper.rb
    require 'serverspec'

    include SpecInfra::Helper::Exec
    include SpecInfra::Helper::DetectOS

    RSpec.configure do |c|
      if ENV['ASK_SUDO_PASSWORD']
        require 'highline/import'
        c.sudo_password = ask("Enter sudo password: ") { |q| q.echo = false }
      else
        c.sudo_password = ENV['SUDO_PASSWORD']
      end
    end

But the spec_helper file in the blog post is:

    # myface/test/integration/default/serverspec/spec_helper.rb
    require 'serverspec'
    require 'pathname'

    include Serverspec::Helper::Exec
    include Serverspec::Helper::DetectOS

    RSpec.configure do |c|
      c.before :all do
        c.os = backend(Serverspec::Commands::Base).check_os
      end
    end

I am going to try a combination. I moved my emacs_spec.rb into spec/localhost/ and then renamed the spec directory to serverspec. I left the includes of the SpecInfra::Helper modules but removed the sudo stuff and replaced it with the check_os stuff from the blog post. So now I have:

    # chef-linode/test/integration/default/serverspec/spec_helper.rb
    require 'serverspec'

    include SpecInfra::Helper::Exec
    include SpecInfra::Helper::DetectOS

    RSpec.configure do |c|
      c.before :all do
        c.os = backend(Serverspec::Commands::Base).check_os
      end
    end

Yeah!!! now my kitchen converge boots, tries to converge (but I have nothing in my runlist) and then runs the test.

If you are using code to set up your infrastructure, then that code demands as good or better software development practices as your application code. One of the reasons I wanted to try Chef is because I knew that people were doing automated testing of chef configurations. (The puppet community may be doing similar kinds of testing; I don’t know.) There is even a book about TDD and Chef: Test-Driven Infrastructure With Chef (and it’s even in its second edition). So I am going to try working through it. The first several chapters are background or things I have sort of done already. But they start to use Opscode’s Hosted Chef and Vagrant in chapter 4.

Setting Up a New Playground

When I did the Quick Start exercises, I downloaded the starter kit which included a .chef directory with a knife configuration file and two .pem files: ckiser.pem and ckiser-validator.pem. I explored downloading the starter kit again - but it wanted to reset my keys which would have made my example stuff not work any more. It took me a little while to figure out the instrutions on http://docs.opscode.com/config_rb_knife.html mean that one or the other file is read - not that one is read and then the second file is read, overwriting configuration variables. I was hoping to put shared information in ~/.chef/knife.rb and then project specific information in a knife.rb in the project directory. From my ‘puts’ statement, only one of the files is read - either the one in the current directory or the one in ~/.chef/.

I can still get part of what I want - reuse of my keys - but moving them to ~/.chef/ and then editing my knife.rb file to look for them there. And then I created a new directory for working the TDI Chef exercises, ~/chef-tdd/. I copied the knife.rb from the starter kit into ~/.chef/knife.rb. I also created a cookbooks directory and copied the chefignore file from the starter kit in there. And added the extensive .gitignore from the starter kit. There are some other things I think I will probably need but am going to wait until something I am doing uses them before mving them here.

Vagrant

The last part of chapter 4 is about installing Vagrant and using it to create virtual machines. Mostly this is just like what I have been doing for a while now - but it linked me to a very promising source for base boxes, Opscode’s Bento Boxes One thing that makes them especially attractive is that the don’t come with Chef preinstalled. Instead they recommend installing the most current Chef using the vagrant-omnibus plugin. From the TDI book:

The plug-in we installed into Vagrant (vagrant plugin install vagrant-omnibus) works with Vagrant boxes that do not have Chef installed, and adds a hook to vagrant up to install Chef using the omnibus package, just as we did in “Exercise 1: Install Chef ” on page 47. This helps keep the Vagrant box slim and as close to upstream as possible, and does not require a fleet of Vagrant boxes to be created with every Chef patch release.

I want to test with a box that matches my Linode VPS as closely as possible so despite the fact I already have an Opscode Ubuntu 12.04 in 32 bit, I downloaded and added the 64 bit version:

    vagrant box add opscode-ubuntu-12.04 \
        http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-12.04_chef-provisionerless.box

    [03:31 PM] (brazen:~/chef-tdd) $ vagrant plugin install vagrant-omnibus
    Installing the 'vagrant-omnibus' plugin. This can take a few minutes...
    Installed the plugin 'vagrant-omnibus (1.2.1)'!

Then in ‘~/chef-tdd/ I did vagrant init opscode-ubuntu-12.04` and then edited the Vagrant file to have the following:

    Vagrant.configure("2") do |config|
      config.vm.box = "opscode-ubuntu-12.04"
      config.vm.box_url = "https://opscode-vm.s3.amazonaws.com/vagrant/opscode_ubuntu-12.04_chef-provisionerless.box"
      config.omnibus.chef_version = :latest
      config.vm.hostname = "vagrant-ubuntu-12-04"
      config.vm.network :forwarded_port, guest: 80, host: 8080
    end

Then I did vagrant up. But I didn’t see the lines I expected regarding installing chef. I tried destroying the VM and then tring vagrant up again - same result. I do have warnings about the Guest Additions not matching the version of VirtualBox that the Opscode base boxes were built against:

    [default] Waiting for VM to boot. This can take a few minutes.
    [default] VM booted and ready for use!
    [default] The guest additions on this VM do not match the installed
    version of VirtualBox! In most cases this is fine, but in rare cases
    it can cause things such as shared folders to not work properly. If
    you see shared folder errors, please update the guest additions within
    the virtual machine and reload your VM.

    Guest Additions Version: 4.3.2
    VirtualBox Version: 4.2
    [default] Setting hostname...

So I downloaded the latest version of VirtualBox, 4.3.6, and tried again. That wasn’t so great. I got the following error message:

    $ vagrant up
    Vagrant has detected that you have a version of VirtualBox installed
    that is not supported. Please install one of the supported versions
    listed below to use Vagrant:
    4.0, 4.1, 4.2

I currently have Vagrant 1.2.2 installed. Looking at vagrantup.com, it looks like the latest is 1.4.3. Let’s see if upgrading Vagrant fixes this. I downloaded the latest .dmg file and ran the installer. Then my next attempt at vagrant up gave me:

    The following plugins were installed with a version of Vagrant that
    had different versions of underlying components. Because these
    component versions were changed (which rarely happens), the plugins
    must be uninstalled and reinstalled.

    To ensure that all the dependencies are properly updated as well it is
    _highly recommended_ to do a `vagrant plugin uninstall` prior to
    reinstalling.

    This message will not go away until all the plugins below are either
    uninstalled or uninstalled then reinstalled.

    The plugins below will not be loaded until they're uninstalled and
    reinstalled:

    vagrant-berkshelf, vagrant-omnibus

I unstalled both plugins. I don’t remember installing vagrant-berkshelf so, for now, I am not going to reinstall it. I only reinstalled vagrant-omnibus (1.2.1). But still no joy: no chef-client or knife when I log into the box. ARRRGGGGGG because I was editing the wrong Vagrantfile! I had put the line about vagrant-omnibus into the file in chef-repo that I had opened for reference. OK let’s destroy and recreate the VM. And now, we get chef:

    04:20 PM] (brazen:~/chef-tdd) $ vagrant up
    Bringing machine 'default' up with 'virtualbox' provider...
    [default] Importing base box 'opscode-ubuntu-12.04'...
    [default] Matching MAC address for NAT networking...
    [default] Setting the name of the VM...
    [default] Clearing any previously set forwarded ports...
    [default] Clearing any previously set network interfaces...
    [default] Preparing network interfaces based on configuration...
    [default] Forwarding ports...
    [default] -- 22 => 2222 (adapter 1)
    [default] -- 80 => 8080 (adapter 1)
    [default] Booting VM...
    [default] Waiting for machine to boot. This may take a few minutes...
    [default] Machine booted and ready!
    [default] Setting hostname...
    [default] Mounting shared folders...
    [default] -- /vagrant
    [default] Installing Chef 11.8.2 Omnibus package...
    [default] Downloading Chef 11.8.2 for ubuntu...
    [default] downloading https://www.opscode.com/chef/metadata?v=11.8.2&prerelease=false&p=ubuntu&pv=12.04&m=x86_64
    [default]   to file /tmp/install.sh.1143/metadata.txt
    [default] trying wget...
    [default] url   https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.8.2-1.ubuntu.12.04_amd64.deb
    md5     3d3b3662830a44eeec71aadc098a4018
    sha256  a5b00a24e68e29a01c7ab9de5cdaf0cc9fd1c889599ad9af70293e5b4de8615c
    [default] downloaded metadata file looks valid...
    [default] downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.8.2-1.ubuntu.12.04.amd64.deb
    [default]   to file /tmp/install.sh.1143/chef_11.8.2_amd64.deb
    [default] trying wget...
    [default] Checksum compare with sha256sum succeeded.
    [default] Installing Chef 11.8.2
    [default] installing with dpkg...
    [default] Selecting previously unselected package chef.
    [default] (Reading database ...
    [default] 54659 files and directories currently installed.)
    [default] Unpacking chef (from .../chef_11.8.2_amd64.deb) ...
    [default] Setting up chef (11.8.2-1.ubuntu.12.04) ...
    [default] Thank you for installing Chef!
    [04:24 PM] (brazen:~/chef-tdd) $ vagrant ssh
    Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.8.0-29-generic x86_64)

     * Documentation:  https://help.ubuntu.com/
    Last login: Tue Nov 26 11:27:55 2013 from 10.0.2.2
    vagrant@vagrant-ubuntu-12-04:~$ chef-client --version
    Chef: 11.8.2

Ch 7.2: Berkshelf

Much of configuration management (and really any software development) is dependency management. Throughout the TDI book we were managing that manually - downloading cookbooks, checking dependencies, downloading more cookbooks. Finally in chapter 7 the author introduces Berkshelf to do the managing for us. So now I am going to need the vagrant-berkshelf plugin that I delayed reinstalling in the section above. vagrant plugin install vagrant-berkshelf. And then the book told me to run berks configure. This creates a default configuration file in ~/.berkshelf/config.json. Not really sure what the implications of some of the items in this are. I’ll have to see as they arise.

Since I didn’t make the stand alone IRC cookbook from chapter 3, I can’t do the berks init stuff. And the discussion of Berkshelf kind of trailed off promising more information about preferred workflows later in the chapter.

Ch 7.3: Application Cookbooks and Test Kitchen

What are application cookbooks and why should we use them?

The application cookbook pattern is characterized by having decided the top-level service that we provide and creating a cookbook for that service. That cookbook wraps all the dependent services that are needed to deliver the top-level service. … This looks a lot like the kind of thing that might be accomplished using a Chef role, but has some significant advantages. First of all, cookbooks can be explicitly versioned and tracked in a way that roles can’t.

If there is a need to alter the behavior of an upstream cookbook, attributes can be set in a recipe, and if functionality needs to be added, tested, or tweaked, this can be achieved by wrapping upstream cookbooks in a manner that looks much like object inheritance. This has the twin advantages again of being testable, but also of avoiding constant fork‐ ing of upstream cookbooks.

The test harness tool of choice - at least of this author - is Test Kitchen. I installed it with gem install test-kitchen. The tests themselves are created using a combination of cucumber, rspec, and leibniz (a library written by the author).

Things were going OK until about page 200 where there were some missing steps - such as where are we putting our specs and how did we get our spec_helper - and for that matter, where did we get rspec since it wasn’t on the list of gems we were told to add to our Gemfile. The errata page has a complaint from someone else who noticed this - but not resolution. I spent some time wandering around, read, asked some questions on the #chef IRC channel. And then with some perspective, went back to the book to see if I could reverse engineer stuff using the code listings.

First, if we are going to run rspec -fd, we need rspec installed. So I added it to the Gemfile and did bundle install. Now I can run rspec --init and that created a spec directory and put a spec_helper.rb file in there. I am kind of unclear about where to put the integration test at the bottom of p 202 so I skipped them and started with the unit test at the top of p 204. I typed in the ‘let’ and the first ‘it’ and then ran the spec. It complained uninitialized constant ChefSpec so I added ‘chefspec’ to my Gemfile and a “require ‘chefspec’” to the top of my default_spec.rb. Then it complained it couldn’t find chef. I know I have chef - but it wasn’t in my Gemfile. So I added it to my Gemfile and reran ‘bundle’. Now I am getting an odd error about about Chef - and a really old version of chef. How did that get in here? And how did I end up with chefspec 0.0.1?

    (brazen:~/chef-tdd/cookbooks/cnk-blog) $ rspec
    /Users/cnk/.rvm/gems/ruby-2.0.0-p353@chef/gems/chef-0.8.10/lib/chef/provider/package/dpkg.rb:26:in `<class:Package>': uninitialized constant Chef::Provider::Package::Apt (NameError)
        from /Users/cnk/.rvm/gems/ruby-2.0.0-p353@chef/gems/chef-0.8.10/lib/chef/provider/package/dpkg.rb:25:in `<class:Provider>'

I created a completely new rvm gemset, removed my Gemfile.lock and did a completely new bundle install. This time I only have chef 11.8.2 along with chef-spec 3.2.0 But even after fiddling with how I required chefspec, I am still getting complaints abotu uninitialized constant ChefSpec::ChefRunner. So I am going to give up on the example in Chapter 7 “Acceptance Testing: Cucumber and Liebniz”.

Ch 7.5: Integration Testing: Test Kitchen with Serverspec

The next section on integration testing does a better job of showing us the steps. But introduces one more layer - using Test Kitchen to manage your Vagrant boxes. It is sort of nice because it gives you one file that declares which OSs you are going to test your cookbook on - and takes care of setting up what one needs to do that. But it is one additional layer AND it means I am swimming in Vagrantfiles - most of which are not used. I had thought I would test my full configuration on a local Vagrant instance, so I created a Vagrantfile at the top of my chef-tdd directory. When I did berks cookbook cnk-blog, that automatically created a Vagrant file. And now when I edited the .kitchen.ym file (which berks also created for me) and ran kitchen create all, that created a Vagrantfile in .kitchen/kitchen-vagrant/default-ubuntu-1204/ I think it is this last Vagrantfile that I am actually seeing when I do kitchen list - since I can’t ssh in when I am in ~/chef-tdd/cookbooks/cnk-blog/ but I can if I am in ~/chef-tdd/cookbooks/cnk-blog/.kitchen/kitchen-vagrant/default-ubuntu-1204

My cookbook doesn’t yet have a run list, but if I run kitchen converge anyway, that installs ruby and chef from the Opscode Omnibus installer. So now, where do I put the tests. I don’t think I want to use BATS unless I have to - I am better at Ruby than bash. So I read that part but then started working through the Serverspec exercises starting at p 241. I created a test file in the magic place (test kitchen figures out it needs busser and then what kinds of tests to run based on the directory names).

    $ mkdir -p test/integration/default/serverspec/localhost
    # create file cnk-blog_spec.rb
    $ cat  test/integration/default/serverspec/localhost/cnk-blog_spec.rb
    require 'spec_helper'

    describe "Cynthia's static html blog site" do
      it 'should have installed apache' do
        expect(package apache2).to be_installed
      end
    end

    $ kitchen verify
    -----> Starting Kitchen (v1.1.1)
    -----> Setting up <default-ubuntu-1204>...
    Fetching: thor-0.18.1.gem (100%)
    Fetching: busser-0.6.0.gem (100%)
    Successfully installed thor-0.18.1
    Successfully installed busser-0.6.0
    2 gems installed
    -----> Setting up Busser
           Creating BUSSER_ROOT in /tmp/busser
           Creating busser binstub
           Plugin serverspec installed (version 0.2.6)
    -----> Running postinstall for serverspec plugin
           Finished setting up <default-ubuntu-1204> (0m22.57s).
    -----> Verifying <default-ubuntu-1204>...
           Suite path directory /tmp/busser/suites does not exist, skipping.
    Uploading /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb (mode=0664)
    -----> Running serverspec test suite
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serversp
    ec/localhost/cnk-blog_spec.rb --color --format documentation
    /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- spec_helper (LoadError)
            from /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
            from /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:1:in `<top (required)>'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `load'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `block in load_spec_files'

            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `each'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `load_spec_files'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/command_line.rb:22:in `run'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:80:in `run'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:17:in `block in autorun'
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb --color --format documentation failed
    Ruby Script[/tmp/busser/gems/gems/busser-serverspec-0.2.6/lib/busser/runner_plugin/../serverspec/runner.rb /tmp/busser/suites/serverspec] exit code was 1
    >>>>>> Verify failed on instance <default-ubuntu-1204>.
    >>>>>> Please see .kitchen/logs/default-ubuntu-1204.log for more details
    >>>>>> ------Exception-------
    >>>>>> Class: Kitchen::ActionFailed
    >>>>>> Message: SSH exited (1) for command: [sh -c 'BUSSER_ROOT="/tmp/busser" GEM_HOME="/tmp/busser/gems" GEM_PATH="/tmp/busser/gems" GEM_CACHE="/tmp/busser/gems/cache" ; export BUSSER_ROOT GEM_HOME GEM_PATH GEM_CACHE; sudo -E /tmp/busser/bin/busser test']
    >>>>>> ----------------------

Oops. I didn’t read carefully and missed creating the helper file in cnk-blog/test/integration/default/serverspec/spec_helper.rb

    require 'serverspec'
    require 'pathname'
    include Serverspec::Helper::Exec
    include Serverspec::Helper::DetectOS

    RSpec.configure do |c|
      c.before :all do
        c.os = backend(Serverspec::Commands::Base).check_os
      end
    end

Now I get properly failing tests - that is failing because I have not written the cookbook code to make them work.

    $ kitchen verify
    -----> Starting Kitchen (v1.1.1)
    -----> Verifying <default-ubuntu-1204>...
           Removing /tmp/busser/suites/serverspec
    Uploading /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb (mode=0664)
    Uploading /tmp/busser/suites/serverspec/spec_helper.rb (mode=0664)
    -----> Running serverspec test suite
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb --color --format documentation

    Cynthia's static html blog site
    No packages found matching apache2.
      should have installed apache (FAILED - 1)
      have enabled the apache service (FAILED - 2)
    httpd: unrecognized service
      be running the apache service (FAILED - 3)
      should listen on port 80 (FAILED - 4)
      should have a virtual host for cnk-blog

    Failures:

      1) Cynthia's static html blog site should have installed apache
         Failure/Error: expect(package 'apache2').to be_installed
           dpkg-query -f '${Status}' -W apache2 | grep '^install ok installed$'
           expected Package "apache2" to be installed
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:5:in `block (2 levels) in <top (required)>'

      2) Cynthia's static html blog site have enabled the apache service
         Failure/Error: expect(service 'httpd').to be_enabled
           ls /etc/rc3.d/ | grep -- '^S..httpd' || grep 'start on' /etc/init/httpd.conf
           grep: /etc/init/httpd.conf: No such file or directory

           expected Service "httpd" to be enabled
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:9:in `block (2 levels) in <top (required)>'

      3) Cynthia's static html blog site be running the apache service
         Failure/Error: expect(service 'httpd').to be_running
           ps aux | grep -w -- httpd | grep -qv grep
           expected Service "httpd" to be running
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:13:in `block (2 levels) in <top (required)>'

      4) Cynthia's static html blog site should listen on port 80
         Failure/Error: expect(port 80).to be_listening
           netstat -tunl | grep -- :80\
           expected Port "80" to be listening
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:17:in `block (2 levels) in <top (required)>'

    Finished in 0.06817 seconds
    5 examples, 4 failures

    Failed examples:

    rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:4 # Cynthia's static html blog site should have installed apache
    rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:8 # Cynthia's static html blog site have enabled the apache service
           rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:12 # Cynthia's static html blog site be running the apache service
           rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:16 # Cynthia's static html blog site should listen on port 80
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb --color --format documentation failed
    Ruby Script[/tmp/busser/gems/gems/busser-serverspec-0.2.6/lib/busser/runner_plugin/../serverspec/runner.rb /tmp/busser/suites/serverspec] exit code was 1
    >>>>>> Verify failed on instance <default-ubuntu-1204>.
    >>>>>> Please see .kitchen/logs/default-ubuntu-1204.log for more details
    >>>>>> ------Exception-------
    >>>>>> Class: Kitchen::ActionFailed
    >>>>>> Message: SSH exited (1) for command: [sh -c 'BUSSER_ROOT="/tmp/busser" GEM_HOME="/tmp/busser/gems" GEM_PATH="/tmp/busser/gems" GEM_CACHE="/tmp/busser/gems/cache" ; export BUSSER_ROOT GEM_HOME GEM_PATH GEM_CACHE; sudo -E /tmp/busser/bin/busser test']
    >>>>>> ----------------------

For Future Reference

1. A tutorial in the form of a git repo: RallySoftware-cookbooks/chef-tutorials

2. An excellent blog post

Chef On Linode - with ‘knife linode’

OpsCode’s VirtualBox VM seems pretty light, sudo du -sh / gives 1.1G. Can I use that image to install stuff on Linode? Or am I better off starting with Linode’s Ubuntu 12.04 LTS image? I think I probably want to start with Linode’s image. While looking for other stuff, I ran across the knife-linode plugin. Looking at the docs on the Opscode site, there seem to be configuration parameters for knife-linode that will let you create a server using one of Linode’s stock images so let’s see if I can use that to create a box.

    gem install knife-linode # Got version 0.2.0

To play with it, I needed to go to the Linode “My Profile” page and create an API key. Then I can list my current servers:

    knife linode server list -A <key>

And delete one:

    knife linode server delete <Linode ID> -A <key>

OK so now can I create one?

    $ knife linode server create -A <key> --linode-datacenter 3 \
      --linode-flavor 1 --linode-image 99 --linode-node-name cnk-ubuntu1204-knife \
      --ssh-user cnk --ssh-password <pass>

    Linode ID: 463372
    Name: cnk-ubuntu1204-knife
    IPs: <ip>,192.168.x.x
    Status: Being Created
    Public IP: <ip>
    User: cnk

    Waiting for sshd
    done
    Boostrapping Chef on <ip>

The user doesn’t seem to have been created. So I can’t ssh in as cnk (and cnk isn’t in /etc/passwd), but I can ssh in as root. And I don’t seem to have any ruby or chef - perhaps because I didn’t choose a distro or template file. Let’s try that again - with --distro chef-full added.

That give similar output (on a new IP). But when I logged in, I still don’t have chef or ruby. In fact, there isn’t anything in /opt at all. Let’s see if I can bootstrap chef by hand.

    $ knife bootstrap <ip> -A <key> --distro chef-full
    Bootstrapping Chef on <ip>
    Failed to authenticate root - trying password auth
    Enter your password: ************

    192.81.134.47 --2014-01-13 04:40:10--  https://www.opscode.com/chef/install.sh
    192.81.134.47 Resolving www.opscode.com (www.opscode.com)... 184.106.28.90
    192.81.134.47 Connecting to www.opscode.com (www.opscode.com)|184.106.28.90|:443... connected.
    192.81.134.47 HTTP request sent, awaiting response... 200 OK
    192.81.134.47 Length: 14101 (14K) [application/x-sh]
    192.81.134.47 Saving to: `STDOUT'
    192.81.134.47
    100%[======================================>] 14,101      --.-K/s   in 0s
    192.81.134.47
    192.81.134.47 2014-01-13 04:40:10 (392 MB/s) - written to stdout [14101/14101]
    192.81.134.47
    192.81.134.47 Downloading Chef 11.8.2 for ubuntu...
    192.81.134.47 downloading https://www.opscode.com/chef/metadata?v=11.8.2&prerelease=false&p=ubuntu&pv=12.04&m=x86_64
    192.81.134.47   to file /tmp/install.sh.3138/metadata.txt
    192.81.134.47 trying wget...
    192.81.134.47 url       https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.8.2-1.ubuntu.12.04_amd64.deb
    192.81.134.47 md5       3d3b3662830a44eeec71aadc098a4018
    192.81.134.47 sha256    a5b00a24e68e29a01c7ab9de5cdaf0cc9fd1c889599ad9af70293e5b4de8615c
    192.81.134.47 downloaded metadata file looks valid...
    192.81.134.47 downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.8.2-1.ubuntu.12.04_amd64.deb
    192.81.134.47   to file /tmp/install.sh.3138/chef_11.8.2_amd64.deb
    192.81.134.47 trying wget...
    192.81.134.47 Checksum compare with sha256sum succeeded.
    192.81.134.47 Installing Chef 11.8.2
    192.81.134.47 installing with dpkg...
    192.81.134.47 Selecting previously unselected package chef.
    (Reading database ... 21184 files and directories currently installed.)
    192.81.134.47 Unpacking chef (from .../chef_11.8.2_amd64.deb) ...
    192.81.134.47 Setting up chef (11.8.2-1.ubuntu.12.04) ...
    192.81.134.47 Thank you for installing Chef!
    192.81.134.47 Starting Chef Client, version 11.8.2
    192.81.134.47 Creating a new client identity for localhost using the validator key.
    192.81.134.47 resolving cookbooks for run list: []
    192.81.134.47 Synchronizing Cookbooks:
    192.81.134.47 Compiling Cookbooks...
    192.81.134.47 [2014-01-13T04:40:34+00:00] WARN: Node localhost has an empty run list.
    192.81.134.47 Converging 0 resources
    192.81.134.47 Chef Client finished, 0 resources updated

That seems a bit better; now we have ruby 1.9.3-p484 in /opt/chef/embedded/bin/ruby. And it came with rubygems 1.8.24, bundler (1.1.5), rake (10.1.0, 0.9.2.2), and chef (11.8.2). There are a couple of gems that have more recent versions but I decided not to try running ‘gem update’ because the installed versions may have been chosen specifically to satisfy chef dependencies.

I’ll probably want to have chef doing my updates but for now, I think I should have the server patched so ran apt-get update and apt-get upgrade by hand for now. (Before I ran the upgrade, I took a list of the packages that were installed originally with dpkg-query -W > /root/original-packages.txt so I would know what the Linode Precise Penguin image came with.)

Connecting my Linode node to Chef server

After installing chef with the knife bootstrap command, I see it tries to converge the new node. However, there is nothing in my run list - and no cookbook repository for this new server. Time to make one - and then put my Linode API configuration into the knife.rb file so I can quit passing it on the command line all the time.

UGH actually knife seems to have created a node on the server for me. Was that during knife linode server create? or during the knife bootstrap command? I can’t really tell from the docs. I should have passed -N <name> to knife linode server create so that my node would not end up named ‘localhost’ (and then when that didn’t do the bootstrapping, again send the name). From what I can tell by Googling, you can’t really rename a node, though there are delete and readd proceedures that end up being quite similar. I think I’ll try this one to get the chef node for my Linode sever named something more sensible than localhost. I skipped step 6 since I didn’t see ‘localhost’ in the /etc/chef/client.rb file. Running chef-client -N linode gave:

    # chef-client -N linode
    Starting Chef Client, version 11.8.2
    Creating a new client identity for linode using the validator key.
    resolving cookbooks for run list: []
    Synchronizing Cookbooks:
    Compiling Cookbooks...
    [2014-01-24T06:58:34+00:00] WARN: Node linode has an empty run list.
    Converging 0 resources
    Chef Client finished, 0 resources updated

And now I see a node named “linode” in both the Opscode web interface and in the output of knife node list. Yeah!

My group at $WORK has decided to start using Django as our web development framework so I am working my way through the tutorial. First off, yeah! there is actually a tutorial! Nice to be using a mature technology with good documentation.

The overview does a nice job of showing off some basics - models (including relationships), routing/urls, views, and templates (including template composition and reuse). Which brings me to my first mental mapping. One basic design pattern for web apps is known as MVC - Model View Controller. So Rails calls three of the directories under ‘app’ models, views, and controllers. There is a fourth directory named helpers, which is an awkward catch-all that tries to take the logic out of the view layer, with variable success. And the urls or routes are not defined within ‘app’ at all and instead are defined in the config directory in ‘config/routes.rb’. Rails reuses the names from MVC but in so doing sometimes ends up with some odd compromises. The Django breakdown, on the other hand, seems to do a nicer job of splitting the controller and view logic into the logic part (named views) and the display part, in templates.

One of the best parts of Ruby is rake and Rails makes extensive use of it for setting up and managing apps. In the Django ecosystem, many similar tasks are performed via the ‘manager.py’ script.

Global

Django: “reusable apps” Rails: “engines” (and to a lesser degree, plugins)

Rails: built in sessions but no authentication or authorization. Django: sessions + authentication. Not sure how much authorization infrastructure comes built in.

Troubleshooting

rails console - great for models. Now better than it was for controllers, routes, helpers. Not sure about views. Find a recent blog post and read up / link.

Django shell:

manage.py shell - great for models. To poke around your views layer (anything that needs the response object), you need to add some power:

    >>> from django.test.utils import setup_test_environment
    >>> setup_test_environment()
    >>> from django.test.client import Client
    >>> client = Client()
    >>> from django.core.urlresolvers import reverse
    >>> response = client.get(reverse('polls:index'))
    >>> response.status_code
    200
    >>> response.content

Models

Both Rails and Django make handling associations a breeze. In Django, for one to many associations, all you need to do is create declare an attribute on the many side (the side that gets the mapping column) that is a ForeignKey, e.g.:

    class Article(models.Model):
        pub_date = models.DateField()
        headline = models.CharField(max_length=200)
        content = models.TextField()
        reporter = models.ForeignKey(Reporter)

And Django sets up the mapping for you. Then you can assign a reporter to an article by saying my_article.reporter = some_reporter. And you can chain methods to get to the reporter’s name: my_article.reporter.full_name. From the reporter side, you can get a list of all her articles by asking for murrow.article_set.all(). Rails is similar except that you don’t use foreign key relationships (not my favorite Rails decision) but instead you declare the relationships explicitly in both models:

    class Aritcle << ActiveRecord::Base
        belongs_to :reporter

    class Reporter << ActiveRecord::Base
        has_many :articles

The Django database query interface is a bit like the new ActiveModel syntax - but with ‘get’ instead of ‘find’.

    # Django provides a rich database lookup API.
    >>> Reporter.objects.get(id=1)
    <Reporter: John Smith>
    >>> Reporter.objects.get(full_name__startswith='John')
    <Reporter: John Smith>
    >>> Reporter.objects.get(full_name__contains='mith')
    <Reporter: John Smith>
    >>> Reporter.objects.all()
    [<Reporter: John Smith>]

Even in the first part of the tutorial, I run up against Python’s ‘explicit is better than implicit’ philosophy. Rails will use the class name to infer what instances of your model should be called. You can get to that value using the human_attribute_name method and you can either override that method in your model (old school) or use Rails internationalization capabilities to override the value in your locale config files. ActiveRecord also provides a default display of each instance which includes all the attributes of an instance. By contrast, in Django you need to define a __unicode__() (or in Python 3 a __str__()) method for each model class so that the shell and the admin interface know what to call things. There doesn’t appear to be a default representation that shows all the data values; I suspect you are expected to use Python’s dir() function, e.g. print dir(my_poll).

Models

In Rails the many side of the relationship is accessed as the plural of it’s name: my_poll.choices. In Django you use type + _set: my_poll.choice_set

Rails scopes (formerly known as named_scopes) let’s you make something that looks like a method that can be used to constructed queries - especially useful if you want to chain several of these conditions together but in a readable way.

Poking around in Django it appears to me that there isn’t such a thing

• unless the people answering questions on StackOverflow are as ignorant as I am about Django’s query interface. The answer appears to be, if it’s simple, just put the logic for the method directly into your filter. If it is hairier, possibly write the SQL yourself. ???

Validations? Cool part of Rails. Is there a Django equivalent? or do you do that validation in the view (aka controller)?

Django admin

Wow! Just wow! Rails scaffold gives you OK CRUD functionality and there are some plugins that help make things somewhat fancier. But the out of the box admin functionality in Django is fabulous: sorting, searching, filtering. The closest thing I know in the Rails ecosystem is ActiveAdmin, which is nice, but deviates somewhat from the Rails standard.

One thing I really like a lot is the “History” tab. I end up wanting to add auditing to a lot of things I write but in several cases I have added acts_as_versioned to tracked the data - but didn’t provide an admin interface for displaying the changes. So, of course, now I want more. Does the Django admin interface provide a way to revert to a specific version? And can it diff within a text area?

Controllers / Views

Similar render and redirect options. Django provides two really handy shortcuts get_object_or_404 and [get_list_or_404]((https://docs.djangoproject.com/en/1.8/topics/http/shortcuts/#get-list-or-404). Handy enough I should create equivalents instead of the controller code I have.

Routing

Rails’ own interpretation of RESTful urls is strongly favored - in code and by the community. URLs all defined in a single routes.rb config file - unless you are using an engine. (For a while engines didn’t really have a mechanism for creating their routes other than by having a generator write something into the routes file at install. But now I think the routing infrastructure will look for routes files in engines that are mounted into your Rails project.

Django’s routing has a master file, urls.py but it is expected that each of your site’s apps will define their own urls and that you will just include those urls into your top level urls.py with a line like url(r'^polls/', include('polls.urls')) - with or without a namespace. The point of a namespace is to avoid name conflicts in the global set of url names. In with the include above, the names in polls.urls will be global - so the url named details in polls.urls will be the details url for the entire app. If you say url(r'^polls/', include('polls.urls', namespace='polls')), then they will not be global and instead can be accessed as polls:details.

Both frameworks encourage you to use the url lookup mechanism for creating links instead of hard coding them. The big win for this is that if your url need to change, you only have to change the appropriate routes.rb or urls.py file. The references to the urls can all stay the same.

    
    Django: <a href="{% url 'polls:detail' poll.id %}">{{ poll.question }}</a>
    Rails: <%= link_to poll.question, poll_url(poll.id) %>
   

In both systems, some people like having even shorter syntax for defining redirects - redirecting to the object. In Rails it is, as usual, ‘convention over configuration’. If you have defined your routes using ‘resources’, then you can just say redirect_to object

In Django, you can get a very similar redirect syntax: return redirect(object) But to make it work, you need to define a get_absolute_url method on your model class. For example:

    class List(models.Model):

        def get_absolute_url(self):
            return reverse('view_list', args=[self.id])

Templates

Every web framework provides some way of creating web pages from modular pieces. In fact defining your design and navigation elements in one place is sometimes the major reason for using a framework in the first place. I still need to read ‘The Rails View’ to see if I am doing this right, but I think the Django views may be a bit more flexible than the normal Rails view processing.

Rails - one big layout file - chose which main layout file by setting the name in the controller. ‘yield some_name’ and ‘content_for some_name’. Helpers - methods to munge display variables.

Django general “extends” mechanism. Named blocks. Filters attached to data using unix-style pipe, “|”.

Helpers

One of the fabulous things about Rails is all the useful helper methods available to you. Some were created for internal use but are super handy for display purposes, e.g. humanize, pluralize, etc. Others are specifically for display. Many other frameworks either had their own versions, or borrowed liberally from Rails.

Concept Rails Django Timezone handling django.utils.timezone

Forms

Code generation is nice - but there is a lot less typing involved in declaring the form using the Django admin tool. Field sets are quite nice - esp with the automatic ‘collapse’ option. I think this is fairly similar to Rails’ ActiveAdmin. (Sad to see that AA uses Devise for it’s authenitcation tool.)

Testing

Both frameworks come from a background of TDD and have pretty good tools for writing tests. Ruby actually has an embaarssment of riches since it seems that everyone has written their own testing DSL.

Default task runs all tests, or you can run parts. In Rails there are tests for each layer. For Django, the tests are broken out per app: manage.py test polls. View tests can assert against the content of the page with ‘contains’ and if you don’t find what you are looking for, you can print out the web page content:

    response = self.client.get(reverse('polls:index'))
    print response.content
    self.assertContains(response, "No polls are available.")


     get :show, :url_parts => @nested_page.url_parts
     puts response.body
     assert_select 'form[action*=page/update]'

Granular testing. Easy to do in rspec_rails. Possible but not so easy in test_unit/minitest. Easy in Django - just append the name of a specifit test to your command line:

    $ ./manage.py test animals # runs all tests in the animals app
    $ ./manage.py test animals.AnimalTestCase # runs tests in one class
    $ ./manage.py test animals.AnimalTestCase.test_animals_can_speak # runs one test method

Database for running tests

Djano: Aside from using a separate database, the test runner will otherwise use all of the same database settings you have in your settings file: ENGINE, USER, HOST, etc. The test database is created by the user specified by USER, so you’ll need to make sure that the given user account has sufficient privileges to create a new database on the system.

Both offer fixtures for setting up data. Rails community mainly moved to factories and the Django tutorial has you create a tiny factory as a helper methods. Are there libraries for factories in wide use?

Overriding settings - esp urls for apps that could be mounted anywhere; common option - use a decorator on the test method on or the entire class: @override_settings(LOGIN_URL=’/other/login/’)

Some useful looking Django test methods: SimpleTestCase.assertFieldOutput TestCase.assertFormError

Django integration with Selenium: LiveServerTestCase

I have been using RVM for several years - straight in the terminal and in a bash shell in emacs. But all of a sudden I started having path problems. I don’t remember having updated my RVM version, though I quite possibly I did before I updated my installed rubies for the latest security patches. Or perhaps there is some odd interaction with the ruby that got installed when I installed the Heroku toolbelt (v 3.2.0) this last week; it appears to have installed it’s own ruby in /usr/local/heroku/ruby/bin/ruby but then added /usr/local/heroku/bin with export PATH="/usr/local/heroku/bin:$PATH" in my .bashrc.

The symptom was that in my bash shell within emacs I was getting warnings about RVM not finding what it expected at the start of my path. I should have copied the error message but it was expecting something like /Users/cnk/.rvm/gems/ruby-2.0.0-p353/bin. Initially I was getting a path starting with /usr/local/heroku/bin because the Heroku Toolbelt install had placed that line at the end of my .bashrc file. But even after having moved the heroku line into my .profile before the rvm line, I was still getting oddities when trying to use knive (installed with the chef gem). I tried the fixed suggested in the warning message: rvm get stable --auto-dotfiles. That gave me version 1.25.14. and made modifications to my dotfiles:

    $ rvm get stable --auto-dotfiles
    Downloading https://get.rvm.io
    Turning on auto dotfiles mode.
    Downloading https://github.com/wayneeseguin/rvm/archive/stable.tar.gz

    Upgrading the RVM installation in /Users/cnk/.rvm/
        Removing rvm PATH line from /Users/cnk/.zshrc.
        Adding rvm PATH line to /Users/cnk/.profile /Users/cnk/.bashrc /Users/cnk/.zshrc.
        Removing rvm loading line from /Users/cnk/.bashrc /Users/cnk/.zlogin.
        Adding rvm loading line to /Users/cnk/.bash_profile /Users/cnk/.zlogin.
    Upgrade of RVM in /Users/cnk/.rvm/ is complete.

That made everything work in the terminal but made it much worse in emacs. With the default rvm setup (source "$HOME/.rvm/scripts/rvm" in my .bash_profile file), rvm was not getting loaded in my emacs shell. type -t rvm was returning ‘file’ instead of ‘function’ and none of my rvm commands would work - no rvm use or rvm gemset list. I fooled around for a bit and sorted out that what I need to get rvm to work in the terminal AND in my emacs shell is to have this line in my .bashrc file NOT in .bash_profile or .profile:

    # Load RVM into a shell session *as a function*
    [[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"

That works fine and does not seem to have any problems even if I create a new shell within an old one.

Other RVM and Emacs Options

There are a couple of more sophisticated ways of using rvm within emacs that I came across while trying to figure out why this had suddenly stopped working. One of these days I should look at them:

• Emacs 24 Rails Development Environment - From scratch to Productive in 5 Minutes
• rvm.el - and the resources listed in the README such as Avdi Grimm’s Emacs Reboot series

Path Warnings Redux

I got the same warning again today - when changing directories from within a shell running in emacs:

    [04:12 PM] (brazen:~/chef-tdd) $ rvm list
    Warning! PATH is not properly set up, '/Users/cnk/.rvm/gems/ruby-2.0.0-p353/bin' is not available,
             usually this is caused by shell initialization files - check them for 'PATH=...' entries,
             it might also help to re-add RVM to your dotfiles: 'rvm get stable --auto-dotfiles',
             to fix temporarily in this shell session run: 'rvm use ruby-2.0.0-p353'.

    rvm rubies

       ruby-1.9.3-p374 [ x86_64 ]
       ruby-1.9.3-p484 [ x86_64 ]
       ruby-2.0.0-p0 [ x86_64 ]
    =* ruby-2.0.0-p353 [ x86_64 ]

    # => - current
    # =* - current && default
    #  * - default

This time a) I remembered to record it and b) I read the full message! I went with the last option: rvm use ruby-2.0.0-p353. Running that command got me the same warning again - but once the command executed, then all was repaired and the rest of the commands work fine, e.g. rvm list, rvm gemset list, etc.