CNK's Blog

Upgrading to Rails 4.2.0

Upgrading Rails Versions: 4.1.8 to 4.2.0

After a lot of interruptions for other work, last week I finally finished recreating my CMS app in Rails 4 - just in time for the release of Rails 4.2! Most of the information I have been reading about 4.2 has focused on new features, adequate record, foreign keys, active job, etc. So I was not really prepared for a number of issues that came up as a result of changes and depreciations. On my first pass, 871 of my 1419 tests failed! So guess I had better take some notes as I go along.

To get started, I changed the Rails version in my Gemfile from 4.1.8 to 4.2.0 and ran bundle upgrade rails and then ran my rspec test suite (rspec-rails 3.1.0 with rspec-core 3.1.2). As I said, a TON of tests failed, including the vast majority of my controller tests.

SSL Requirement

All (or nearly all) of the controller test failures pointed to my SSL-requirement gem. I need to go back and diagnose that - but since I can’t easily actually run SSL on my dev server right now, I commented out the gem in my Gemfile and include lines in my application controller:

    diff --git a/Gemfile b/Gemfile
    -gem 'bartt-ssl_requirement', :require => 'ssl_requirement'
    +# gem 'bartt-ssl_requirement', :require => 'ssl_requirement'

    diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb

    -  include ::SslRequirement
    -  ssl_allowed :all
    +  # include ::SslRequirement
    +  # ssl_allowed :all

Deprecation Warnings

New Configuration Options

Each time I run my tests, I was getting a couple of deprecation warnings, one about renaming a configuration option serve_static_assets to serve_static_files and one about suppressing errors raised within after_rollback and after_commit callbacks. The first one was fairly easy to deal with. I just searched my config directory and changed the option name in my test.rb and production.rb files. It isn’t set in development.rb so I continued to leave it out.

For the second warning, I tried adding an option in each of my 3 environments files - setting the option to true for dev and test but keeping the old exception suppression by setting this to false in my production.rb file.

    # Don't suppress errors in after_rollback and after_commit callbacks
    config.active_record.raise_in_transactional_callbacks = true

Unfortunately, setting those options did not fully suppress the deprecation warnings. I am still getting them for all 4 of the models I have that use paperclip (my documents model and all 3 of my image type models). Before reporting an issue, I thought I would try upgrading to the latest version of paperclip. Running bundle update paperclip upgraded me from paperclip 4.2.0 to 4.2.1 and cocaine 0.5.4 to 0.5.5. And that did remove the remaining ‘raise_in_transactional_callbacks’ warnings when I run my tests!!!

Truthiness

One of my tests had several instances of the following warning:

    DEPRECATION WARNING: You attempted to assign a value which is not
    explicitly `true` or `false` to a boolean column. Currently this value
    casts to `false`. This will change to match Ruby's semantics, and will
    cast to `true` in Rails 5. If you would like to maintain the current
    behavior, you should explicitly handle the values you would like cast
    to `false`.

All of these were coming from sections of the test where I was doing a create or update with invalid parameters. I had not given much thought to those params but assigned a nonsense value to one of my boolean columns:

    let(:invalid_attributes) { {:campus_only => 'banana'} }

The model in question is not super picky, so it is a little difficult to find parameters that would invalidate the model. For my create actions, sending in an empty file attribute would invalidate the model object. But for an update, an empty file would be perfectly legitimate since one can update other attributes without uploading a new file. But, since I am mocking out the save action to return false in my ‘update with invalid params’ tests, it is good enough.

Page Versions

When running the pages controller spec, I got a bunch of warnings about serialized_attributes. I suspect those are coming from my versioning system, paper_trail. But I noticed that there is also a new version of the tree plugin I use, closure tree. Looking at the changelog, I didn’t see anything that looked like it would make any difference but decided to upgrade anyway so as to keep up; this took me from closure_tree 5.0.0 to 5.2.0.

Upgrading paper_trail from 3.0.5 to 3.0.6 did not make any difference in the serialized_attributes warnings:

    DEPRECATION WARNING: `serialized_attributes` is deprecated without
    replacement, and will be removed in Rails 5.0. (called from block in
    update at /data1/srv/rails/empcms4/app/controllers/pages_controller.rb:62)

Looking at the issue queue, I guess I should have known about this for quite a while: https://github.com/airblade/paper_trail/issues/416 There doesn’t seem to be any consensus about a solution. For now I think I’ll ignore it - even though it means my page controller and versions controller specs are kind of noisy.

Errors

Changes to assert_select

Almost all of my request specs and a bunch of my view specs are failing with messages like:

    NotImplementedError:
       Implementing document_root_element makes assert_select work
       without needing to specify an element to select from.

Mass Upgrade of Gems

Fiddling with a few gem updates didn’t seem to be making any difference to this, so I decided to move my Gemfile.lock to Gemfile.lock.bak (for reference) and let bundler install the latest compatible versions of things. This made the following updates:

    acts_as_list           0.4.0        0.6.0
    aws-sdk                1.55.0       1.60.2
    columnize              0.8.9        0.9.0
    execjs                 2.2.1        2.2.2
    factory_girl_rails     4.4.0        4.5.0
    factory_girl           4.4.0        4.5.0
    jquery-rails           3.1.2        4.0.2
    jquery-ui-rails        5.0.0        5.0.3
    libv8                  3.16.14.5    3.16.14.7
    mini_magick            3.8.1        4.0.1
    mini_portile           0.6.0        0.6.1
    mysql2                 0.3.16       0.3.17
    nokogiri               1.6.3.1      1.6.5
    quiet_assets           1.0.3        1.1.0       (dev and test only)
    rspec-core             3.1.2        3.1.7
    rspec-expectations     3.1.0        3.1.2
    rspec-mocks            3.1.0        3.1.3
    rspec-support          3.1.0        3.1.2
    safe_yaml              1.0.3        1.0.4
    sass-rails             4.0.3        4.0.5
    sass                   3.2.0        3.2.2
    spring                 1.1.3        1.2.0
    sprockets              2.11.0       2.12.3
    uglifier               2.5.3        2.6.0
    webmock                1.18.0       1.20.4

I also commented out coffee-rails because I am not using coffeescript; this removed the coffee-rails, coffee-script-source, and subexec gems.

This upgrade took care of most of the errors I was seeing in my view specs. The few remaining problems pointed out actual errors - mostly some form tags that had been copied over from the rails 2.3 version of this project. I also had an error in the user mailer - again due to misuse of url_for to get an https url. I fixed those and now all of my model, mailer, routing, controller, helper, and view tests pass. So now it is just my request specs that are dorked:

    $ rspec spec/requests
    Finished in 11.54 seconds (files took 4.56 seconds to load)
    146 examples, 123 failures, 3 pending

Defining document_root_element

All the remaining errors still refer to document_root_element not being implemented. Grepping through the gems in my Rails 4.2 gemset, most references come from a new gem, rails-dom-testing. The README for the gem shows an example of using document_root_element - and refers us to the API docs in selector_assertions.rb. But even after reading that, I am somewhat unclear about what to do.

More poking around - by searching on github - turned up 2 pull requests: https://github.com/rails/rails/pull/17107 and https://github.com/rails/rails-dom-testing/pull/20 Again, very confusing but suggesting the need to add an alias to html_document - temporarily?

I looked at the other places where I found document_root_element in my rails 4.2 gemset. All 3 were places where a method by that name was defined, one each in actionpack-4.2.0/lib/action_controller/test_case.rb, actionpack-4.2.0/lib/action_dispatch/testing/integration.rb, and actionview-4.2.0/lib/action_view/test_case.rb. Two out of 3 of those defined it as:

    def document_root_element
      html_document.root
    end

So I tried dropping that into one of my test files and that caused the tests in that file to pass.

To include this in all my request specs, I created a module and included it only in my request specs:

    # spec/support/assert_select_root.rb

    module AssertSelectRoot
      def document_root_element
        html_document.root
      end
    end

    RSpec.configure do |config|
        config.include AssertSelectRoot, :type => :request
    end

With that addition, almost all of my request specs passed. I had a couple of places (in request specs and in my view specs) where I needed to change my html entities to the characters they evaluate to - as warned about in the release notes: http://edgeguides.rubyonrails.org/4_2_release_notes.html#assert-select After taking care of those issues, all my tests pass again!

Other

I still haven’t figured out the SSL requirement errors. But that is for another post.

After that, I may still want to look into rest of the release notes to see if there is anything else I should examine: And I should read through the upgrade guide to see what else I might want to deal with. There is also an intro to Rails 4.2 video

  • though I suspect that will be about the new features and not about some of the refactoring or internal changes.

Migrating to Rails 4.1 - Part-2

Asset Pipeline

OK after a Pythonic interuption and a great vacation, back to my Rails upgrade. The original project was built way before the introduction of the asset pipeline. We we did the last rebranding, I was going cross-eyed trying to find all the references to colors - esp. as there were a couple of itterations from the design firm about exactly which gray was what. So I made myself a global color and font variable list and started to use less in development - with Less.app to recompile the stylesheets on the fly. I could just continue doing that - but it seems a shame not to use one of the big improvements in the Rails ecosystem. And we could use some help to get better browser/proxy caching and reliable cache expiration upon update.

So, I copied my *.less files from ‘public/stylesheets/’ into ‘app/assets/stylesheets’, renamed them from ‘.less’ to ‘.css.scss’, and updated all the variable references from @thing to $thing. Since I want my global variables used everywhere, I want a global compilation scope. According to the ROR Guide to the asset pipeline, that means I should use SASS’s @import directives rather than the rails asset pipeline require mechanism. I thought I understood - but then I was just getting an application.css file with the naked ‘@import “whatever”’ lines. The part I had overlooked was changing ‘application.css’ to ‘application.sass’. Once I did that, I got my content - with the variables correctly interpolated. I was still missing a couple of images referenced in the stylesheets. To fix that, all I needed to do was change from hand coding the url attribute background: url('/images/mte/search_button.png') no-repeat; to using the image-url helper background: image-url("mte/search_button.png") no-repeat;.

Getting my fonts to show up was a little more confusing. I started with a blog post from Atomic Object. It was for Rails 3.1 but I don’t think there have been major changes to the asset pipeline for Rails 4.1 - and the css files they generated from Font Squirrel look a lot like the files we got from the design firm. In the current site, I have all the font information in /public/fonts. I initially tried moving that directory into /app/assets/ but I had trouble finding an import statement that got the SASS compiler to include them in the compiled application.css file; using a relative url @import "../fonts/Caslon" got SASS to include them. But then I had trouble with the font-url helper - couldn’t figure out how to specify the location of the actual font files. The easiest working combination seems to be to move the fonts directory to app/assets/stylesheets/fonts, import Vaud.css.scss using @import "fonts/Vaud", and then use font-urls like src: font-url('fonts/VaudBook-webfont.eot'); /* IE9 Compat Modes */.

Migrating to Rails 4.1 - Part 1

For a variety of reasons we postponed upgrading our custom CMS from Rails 2 to 3 for …. quite a while. Migrating on the bleeding edge is hard - but so is being so far behind. In particular, the advice to upgrade step by step, using the rails rake tasks to automate some things, doesn’t really work because it is too difficult to figure out compatible sets of plugins. For one thing, my testing frameworks have undergone drastic changes. So even if I can automate upgrading rails, I can’t run the tests to verify the upgrades. So I am going to just start over with the current version of rails: 4.1.0.

Using RVM, ruby-1.9.3-p484, and a fresh gemset (empcms4), I generated a new project. I don’t think turbolinks is going to be especially useful, so I removed it. I really like shoulda, so that’s a must. I am less adamant about test framework. I generally prefer the minitest assertion syntax but I just spent a frustrating hour trying to use MiniTest::Mock to get a controller destroy action to claim it failed so I could test the responder from Chapter 6 of Crafting Rails 4 Applications. So when I read that Thoughtbot uses shoulda + rspec, I decided to use rspec.

This blog post, Setting up the BDD stack on a new Rails 4 application, has some pretty good advise on how to set up the things I want to use. Remember to include ‘rspec-rails’ in both the test and development groups so generators will create rspec tests for things.

I regenerated the current pages model and added some not null constraints and default values. I like using FactoryGirl and have a number of utility methods I would like to transfer from my original codebase. I installed factory_girl_rails and then configured it as described in the Getting Started docs.

Chef Concepts

The best - and sometimes the worst - thing about Chef is its thriving ecosystem. There appear to be cookbooks for just about everything, including one just for installing emacs. And there are lots of tools, lots of instructions, and blog posts about using the tools. So many in fact that it is sometimes hard to sort out what is what - especially as many of them are focused on showing you an example project rather than talking about broad concepts. I have picked up a number of useful tidbits but it’s been a little tough to integrate everything.

So, let me start with what I think I want. I want to use chef to set up my Linode VPS. Before I make changes to the server, I would like to run the same recipes locally on a Vagrant VM and run some integration tests to check that I am going to get what I intend. Eventually I may want to add unit tests for my custom cookbooks but for now, I want to just focus on ‘outside in’ testing which, in this situation, seems to me to be integration tests. This blog post, Getting Started Writing Chef Cookbooks the Berkshelf Way, Part 3, appears to do what I want to do, so let’s give that a try.

Initialize

I was originally going to manage everything using chef solo because I thought that would be easier. But it turns out that chef and all the tooling around it are optimized for using chef server to manage lots of nodes, so chef server is actually the easier option - especially since Opscode has a free tier for managing up to five nodes.

When I created my Linode VM with knife-linode I did it all on the command line - no files. So that means that while I have a node named ‘linode’ on my hosted chef and my Linode VPS is set up as a client of that node, I don’t have any files and no git repository keeping track of anything. So let’s see if I can construct something to connect to that node on https://manage.opscode.com/. So I stared with the chef gemset I had previously set up and upgraded (chef 11.10.0, kitchen-vagrant 0.14.0, test-kitchen 1.1.1).

    $ mkdir ~/chef-linode
    $ cd chef-linode
    $ kitchen init
          create  .kitchen.yml
          create  test/integration/default
             run  gem install kitchen-vagrant from "."
    Successfully installed kitchen-vagrant-0.14.0
    Parsing documentation for kitchen-vagrant-0.14.0
    Installing ri documentation for kitchen-vagrant-0.14.0
    1 gem installed

kitchen init created the .kitchen.yml file which is used to configure my test kithcet setup. I added the port forwarding I will need once I have a web server running on my VM. It is configured to spin up a Vagrant VM with the Opscode Ubuntu 12.04 provisionless VM and then use chef_omnibus to install the latest version of chef and provision with chef_solo. I am not sure whether I should be using chef_solo (so I can test out my cookbooks before they get uploaded to Opscode) or if I should change that configuration to point to the linode node on my chef server.

First server spec

So based on the example I worked through from the TDD book, and some other blog posts, I created a spec file in chef-linode/test/integration/default/serverspec/emacs_spec.rb

    require 'spec_helper'

    describe "emacs cookbook" do
      it "should have installed emacs" do
        expect(package 'emacs23-nox').to be_installed
      end
    end

I think this should indicate to test kitchen that I want it to use serverspec to check that I have installed emacs on the VM. Right now, I don’t have a spec_helper file; according to the Getting Started section, I get one by doing serverspec-init. But I am a little unclear about where to put it. The serverspec docs indicate I should have the directory a directory structure like:

    |-- spec
    |   |-- spec_helper.rb
    |   |-- <servername> # e.g localhost or cynthiakiser.com
    |       |-- emacs_spec.rb

But the blog post I am following, shows a directory structure of:

    |-- serverspec
    |   |-- spec_helper.rb
    |   |-- <servername> # e.g localhost or cynthiakiser.com
    |       |-- emacs_spec.rb

And I need a spec_helper.rb. serverspec-init will generate one:

    (brazen:~/chef-linode/test/integration/default) $ serverspec-init
    Select OS type:

      1) UN*X
      2) Windows

    Select number: 1

    Select a backend type:

      1) SSH
      2) Exec (local)

    Select number: 2

     + spec/
     + spec/localhost/
     + spec/localhost/httpd_spec.rb
     + spec/spec_helper.rb
     + Rakefile


    (brazen:~/chef-linode/test/integration/default) $ cat serverspec/spec_helper.rb
    require 'serverspec'

    include SpecInfra::Helper::Exec
    include SpecInfra::Helper::DetectOS

    RSpec.configure do |c|
      if ENV['ASK_SUDO_PASSWORD']
        require 'highline/import'
        c.sudo_password = ask("Enter sudo password: ") { |q| q.echo = false }
      else
        c.sudo_password = ENV['SUDO_PASSWORD']
      end
    end

But the spec_helper file in the blog post is:

    # myface/test/integration/default/serverspec/spec_helper.rb
    require 'serverspec'
    require 'pathname'

    include Serverspec::Helper::Exec
    include Serverspec::Helper::DetectOS

    RSpec.configure do |c|
      c.before :all do
        c.os = backend(Serverspec::Commands::Base).check_os
      end
    end

I am going to try a combination. I moved my emacs_spec.rb into spec/localhost/ and then renamed the spec directory to serverspec. I left the includes of the SpecInfra::Helper modules but removed the sudo stuff and replaced it with the check_os stuff from the blog post. So now I have:

    # chef-linode/test/integration/default/serverspec/spec_helper.rb
    require 'serverspec'

    include SpecInfra::Helper::Exec
    include SpecInfra::Helper::DetectOS

    RSpec.configure do |c|
      c.before :all do
        c.os = backend(Serverspec::Commands::Base).check_os
      end
    end

Yeah!!! now my kitchen converge boots, tries to converge (but I have nothing in my runlist) and then runs the test.

Test-Driven Infrastructure With Chef

If you are using code to set up your infrastructure, then that code demands as good or better software development practices as your application code. One of the reasons I wanted to try Chef is because I knew that people were doing automated testing of chef configurations. (The puppet community may be doing similar kinds of testing; I don’t know.) There is even a book about TDD and Chef: Test-Driven Infrastructure With Chef (and it’s even in its second edition). So I am going to try working through it. The first several chapters are background or things I have sort of done already. But they start to use Opscode’s Hosted Chef and Vagrant in chapter 4.

Setting Up a New Playground

When I did the Quick Start exercises, I downloaded the starter kit which included a .chef directory with a knife configuration file and two .pem files: ckiser.pem and ckiser-validator.pem. I explored downloading the starter kit again - but it wanted to reset my keys which would have made my example stuff not work any more. It took me a little while to figure out the instrutions on http://docs.opscode.com/config_rb_knife.html mean that one or the other file is read - not that one is read and then the second file is read, overwriting configuration variables. I was hoping to put shared information in ~/.chef/knife.rb and then project specific information in a knife.rb in the project directory. From my ‘puts’ statement, only one of the files is read - either the one in the current directory or the one in ~/.chef/.

I can still get part of what I want - reuse of my keys - but moving them to ~/.chef/ and then editing my knife.rb file to look for them there. And then I created a new directory for working the TDI Chef exercises, ~/chef-tdd/. I copied the knife.rb from the starter kit into ~/.chef/knife.rb. I also created a cookbooks directory and copied the chefignore file from the starter kit in there. And added the extensive .gitignore from the starter kit. There are some other things I think I will probably need but am going to wait until something I am doing uses them before mving them here.

Vagrant

The last part of chapter 4 is about installing Vagrant and using it to create virtual machines. Mostly this is just like what I have been doing for a while now - but it linked me to a very promising source for base boxes, Opscode’s Bento Boxes One thing that makes them especially attractive is that the don’t come with Chef preinstalled. Instead they recommend installing the most current Chef using the vagrant-omnibus plugin. From the TDI book:

The plug-in we installed into Vagrant (vagrant plugin install vagrant-omnibus) works with Vagrant boxes that do not have Chef installed, and adds a hook to vagrant up to install Chef using the omnibus package, just as we did in “Exercise 1: Install Chef ” on page 47. This helps keep the Vagrant box slim and as close to upstream as possible, and does not require a fleet of Vagrant boxes to be created with every Chef patch release.

I want to test with a box that matches my Linode VPS as closely as possible so despite the fact I already have an Opscode Ubuntu 12.04 in 32 bit, I downloaded and added the 64 bit version:

    vagrant box add opscode-ubuntu-12.04 \
        http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-12.04_chef-provisionerless.box

    [03:31 PM] (brazen:~/chef-tdd) $ vagrant plugin install vagrant-omnibus
    Installing the 'vagrant-omnibus' plugin. This can take a few minutes...
    Installed the plugin 'vagrant-omnibus (1.2.1)'!

Then in ‘~/chef-tdd/ I did vagrant init opscode-ubuntu-12.04` and then edited the Vagrant file to have the following:

    Vagrant.configure("2") do |config|
      config.vm.box = "opscode-ubuntu-12.04"
      config.vm.box_url = "https://opscode-vm.s3.amazonaws.com/vagrant/opscode_ubuntu-12.04_chef-provisionerless.box"
      config.omnibus.chef_version = :latest
      config.vm.hostname = "vagrant-ubuntu-12-04"
      config.vm.network :forwarded_port, guest: 80, host: 8080
    end

Then I did vagrant up. But I didn’t see the lines I expected regarding installing chef. I tried destroying the VM and then tring vagrant up again - same result. I do have warnings about the Guest Additions not matching the version of VirtualBox that the Opscode base boxes were built against:

    [default] Waiting for VM to boot. This can take a few minutes.
    [default] VM booted and ready for use!
    [default] The guest additions on this VM do not match the installed
    version of VirtualBox! In most cases this is fine, but in rare cases
    it can cause things such as shared folders to not work properly. If
    you see shared folder errors, please update the guest additions within
    the virtual machine and reload your VM.

    Guest Additions Version: 4.3.2
    VirtualBox Version: 4.2
    [default] Setting hostname...

So I downloaded the latest version of VirtualBox, 4.3.6, and tried again. That wasn’t so great. I got the following error message:

    $ vagrant up
    Vagrant has detected that you have a version of VirtualBox installed
    that is not supported. Please install one of the supported versions
    listed below to use Vagrant:
    4.0, 4.1, 4.2

I currently have Vagrant 1.2.2 installed. Looking at vagrantup.com, it looks like the latest is 1.4.3. Let’s see if upgrading Vagrant fixes this. I downloaded the latest .dmg file and ran the installer. Then my next attempt at vagrant up gave me:

    The following plugins were installed with a version of Vagrant that
    had different versions of underlying components. Because these
    component versions were changed (which rarely happens), the plugins
    must be uninstalled and reinstalled.

    To ensure that all the dependencies are properly updated as well it is
    _highly recommended_ to do a `vagrant plugin uninstall` prior to
    reinstalling.

    This message will not go away until all the plugins below are either
    uninstalled or uninstalled then reinstalled.

    The plugins below will not be loaded until they're uninstalled and
    reinstalled:

    vagrant-berkshelf, vagrant-omnibus

I unstalled both plugins. I don’t remember installing vagrant-berkshelf so, for now, I am not going to reinstall it. I only reinstalled vagrant-omnibus (1.2.1). But still no joy: no chef-client or knife when I log into the box. ARRRGGGGGG because I was editing the wrong Vagrantfile! I had put the line about vagrant-omnibus into the file in chef-repo that I had opened for reference. OK let’s destroy and recreate the VM. And now, we get chef:

    04:20 PM] (brazen:~/chef-tdd) $ vagrant up
    Bringing machine 'default' up with 'virtualbox' provider...
    [default] Importing base box 'opscode-ubuntu-12.04'...
    [default] Matching MAC address for NAT networking...
    [default] Setting the name of the VM...
    [default] Clearing any previously set forwarded ports...
    [default] Clearing any previously set network interfaces...
    [default] Preparing network interfaces based on configuration...
    [default] Forwarding ports...
    [default] -- 22 => 2222 (adapter 1)
    [default] -- 80 => 8080 (adapter 1)
    [default] Booting VM...
    [default] Waiting for machine to boot. This may take a few minutes...
    [default] Machine booted and ready!
    [default] Setting hostname...
    [default] Mounting shared folders...
    [default] -- /vagrant
    [default] Installing Chef 11.8.2 Omnibus package...
    [default] Downloading Chef 11.8.2 for ubuntu...
    [default] downloading https://www.opscode.com/chef/metadata?v=11.8.2&prerelease=false&p=ubuntu&pv=12.04&m=x86_64
    [default]   to file /tmp/install.sh.1143/metadata.txt
    [default] trying wget...
    [default] url   https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.8.2-1.ubuntu.12.04_amd64.deb
    md5     3d3b3662830a44eeec71aadc098a4018
    sha256  a5b00a24e68e29a01c7ab9de5cdaf0cc9fd1c889599ad9af70293e5b4de8615c
    [default] downloaded metadata file looks valid...
    [default] downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.8.2-1.ubuntu.12.04.amd64.deb
    [default]   to file /tmp/install.sh.1143/chef_11.8.2_amd64.deb
    [default] trying wget...
    [default] Checksum compare with sha256sum succeeded.
    [default] Installing Chef 11.8.2
    [default] installing with dpkg...
    [default] Selecting previously unselected package chef.
    [default] (Reading database ...
    [default] 54659 files and directories currently installed.)
    [default] Unpacking chef (from .../chef_11.8.2_amd64.deb) ...
    [default] Setting up chef (11.8.2-1.ubuntu.12.04) ...
    [default] Thank you for installing Chef!
    [04:24 PM] (brazen:~/chef-tdd) $ vagrant ssh
    Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.8.0-29-generic x86_64)

     * Documentation:  https://help.ubuntu.com/
    Last login: Tue Nov 26 11:27:55 2013 from 10.0.2.2
    vagrant@vagrant-ubuntu-12-04:~$ chef-client --version
    Chef: 11.8.2

Ch 7.2: Berkshelf

Much of configuration management (and really any software development) is dependency management. Throughout the TDI book we were managing that manually - downloading cookbooks, checking dependencies, downloading more cookbooks. Finally in chapter 7 the author introduces Berkshelf to do the managing for us. So now I am going to need the vagrant-berkshelf plugin that I delayed reinstalling in the section above. vagrant plugin install vagrant-berkshelf. And then the book told me to run berks configure. This creates a default configuration file in ~/.berkshelf/config.json. Not really sure what the implications of some of the items in this are. I’ll have to see as they arise.

Since I didn’t make the stand alone IRC cookbook from chapter 3, I can’t do the berks init stuff. And the discussion of Berkshelf kind of trailed off promising more information about preferred workflows later in the chapter.

Ch 7.3: Application Cookbooks and Test Kitchen

What are application cookbooks and why should we use them?

The application cookbook pattern is characterized by having decided the top-level service that we provide and creating a cookbook for that service. That cookbook wraps all the dependent services that are needed to deliver the top-level service. … This looks a lot like the kind of thing that might be accomplished using a Chef role, but has some significant advantages. First of all, cookbooks can be explicitly versioned and tracked in a way that roles can’t.

If there is a need to alter the behavior of an upstream cookbook, attributes can be set in a recipe, and if functionality needs to be added, tested, or tweaked, this can be achieved by wrapping upstream cookbooks in a manner that looks much like object inheritance. This has the twin advantages again of being testable, but also of avoiding constant fork‐ ing of upstream cookbooks.

The test harness tool of choice - at least of this author - is Test Kitchen. I installed it with gem install test-kitchen. The tests themselves are created using a combination of cucumber, rspec, and leibniz (a library written by the author).

Things were going OK until about page 200 where there were some missing steps - such as where are we putting our specs and how did we get our spec_helper - and for that matter, where did we get rspec since it wasn’t on the list of gems we were told to add to our Gemfile. The errata page has a complaint from someone else who noticed this - but not resolution. I spent some time wandering around, read, asked some questions on the #chef IRC channel. And then with some perspective, went back to the book to see if I could reverse engineer stuff using the code listings.

First, if we are going to run rspec -fd, we need rspec installed. So I added it to the Gemfile and did bundle install. Now I can run rspec --init and that created a spec directory and put a spec_helper.rb file in there. I am kind of unclear about where to put the integration test at the bottom of p 202 so I skipped them and started with the unit test at the top of p 204. I typed in the ‘let’ and the first ‘it’ and then ran the spec. It complained uninitialized constant ChefSpec so I added ‘chefspec’ to my Gemfile and a “require ‘chefspec’” to the top of my default_spec.rb. Then it complained it couldn’t find chef. I know I have chef - but it wasn’t in my Gemfile. So I added it to my Gemfile and reran ‘bundle’. Now I am getting an odd error about about Chef - and a really old version of chef. How did that get in here? And how did I end up with chefspec 0.0.1?

    (brazen:~/chef-tdd/cookbooks/cnk-blog) $ rspec
    /Users/cnk/.rvm/gems/ruby-2.0.0-p353@chef/gems/chef-0.8.10/lib/chef/provider/package/dpkg.rb:26:in `<class:Package>': uninitialized constant Chef::Provider::Package::Apt (NameError)
        from /Users/cnk/.rvm/gems/ruby-2.0.0-p353@chef/gems/chef-0.8.10/lib/chef/provider/package/dpkg.rb:25:in `<class:Provider>'

I created a completely new rvm gemset, removed my Gemfile.lock and did a completely new bundle install. This time I only have chef 11.8.2 along with chef-spec 3.2.0 But even after fiddling with how I required chefspec, I am still getting complaints abotu uninitialized constant ChefSpec::ChefRunner. So I am going to give up on the example in Chapter 7 “Acceptance Testing: Cucumber and Liebniz”.

Ch 7.5: Integration Testing: Test Kitchen with Serverspec

The next section on integration testing does a better job of showing us the steps. But introduces one more layer - using Test Kitchen to manage your Vagrant boxes. It is sort of nice because it gives you one file that declares which OSs you are going to test your cookbook on - and takes care of setting up what one needs to do that. But it is one additional layer AND it means I am swimming in Vagrantfiles - most of which are not used. I had thought I would test my full configuration on a local Vagrant instance, so I created a Vagrantfile at the top of my chef-tdd directory. When I did berks cookbook cnk-blog, that automatically created a Vagrant file. And now when I edited the .kitchen.ym file (which berks also created for me) and ran kitchen create all, that created a Vagrantfile in .kitchen/kitchen-vagrant/default-ubuntu-1204/ I think it is this last Vagrantfile that I am actually seeing when I do kitchen list - since I can’t ssh in when I am in ~/chef-tdd/cookbooks/cnk-blog/ but I can if I am in ~/chef-tdd/cookbooks/cnk-blog/.kitchen/kitchen-vagrant/default-ubuntu-1204

My cookbook doesn’t yet have a run list, but if I run kitchen converge anyway, that installs ruby and chef from the Opscode Omnibus installer. So now, where do I put the tests. I don’t think I want to use BATS unless I have to - I am better at Ruby than bash. So I read that part but then started working through the Serverspec exercises starting at p 241. I created a test file in the magic place (test kitchen figures out it needs busser and then what kinds of tests to run based on the directory names).

    $ mkdir -p test/integration/default/serverspec/localhost
    # create file cnk-blog_spec.rb
    $ cat  test/integration/default/serverspec/localhost/cnk-blog_spec.rb
    require 'spec_helper'

    describe "Cynthia's static html blog site" do
      it 'should have installed apache' do
        expect(package apache2).to be_installed
      end
    end

    $ kitchen verify
    -----> Starting Kitchen (v1.1.1)
    -----> Setting up <default-ubuntu-1204>...
    Fetching: thor-0.18.1.gem (100%)
    Fetching: busser-0.6.0.gem (100%)
    Successfully installed thor-0.18.1
    Successfully installed busser-0.6.0
    2 gems installed
    -----> Setting up Busser
           Creating BUSSER_ROOT in /tmp/busser
           Creating busser binstub
           Plugin serverspec installed (version 0.2.6)
    -----> Running postinstall for serverspec plugin
           Finished setting up <default-ubuntu-1204> (0m22.57s).
    -----> Verifying <default-ubuntu-1204>...
           Suite path directory /tmp/busser/suites does not exist, skipping.
    Uploading /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb (mode=0664)
    -----> Running serverspec test suite
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serversp
    ec/localhost/cnk-blog_spec.rb --color --format documentation
    /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- spec_helper (LoadError)
            from /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
            from /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:1:in `<top (required)>'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `load'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `block in load_spec_files'

            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `each'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `load_spec_files'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/command_line.rb:22:in `run'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:80:in `run'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:17:in `block in autorun'
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb --color --format documentation failed
    Ruby Script[/tmp/busser/gems/gems/busser-serverspec-0.2.6/lib/busser/runner_plugin/../serverspec/runner.rb /tmp/busser/suites/serverspec] exit code was 1
    >>>>>> Verify failed on instance <default-ubuntu-1204>.
    >>>>>> Please see .kitchen/logs/default-ubuntu-1204.log for more details
    >>>>>> ------Exception-------
    >>>>>> Class: Kitchen::ActionFailed
    >>>>>> Message: SSH exited (1) for command: [sh -c 'BUSSER_ROOT="/tmp/busser" GEM_HOME="/tmp/busser/gems" GEM_PATH="/tmp/busser/gems" GEM_CACHE="/tmp/busser/gems/cache" ; export BUSSER_ROOT GEM_HOME GEM_PATH GEM_CACHE; sudo -E /tmp/busser/bin/busser test']
    >>>>>> ----------------------

Oops. I didn’t read carefully and missed creating the helper file in cnk-blog/test/integration/default/serverspec/spec_helper.rb

    require 'serverspec'
    require 'pathname'
    include Serverspec::Helper::Exec
    include Serverspec::Helper::DetectOS

    RSpec.configure do |c|
      c.before :all do
        c.os = backend(Serverspec::Commands::Base).check_os
      end
    end

Now I get properly failing tests - that is failing because I have not written the cookbook code to make them work.

    $ kitchen verify
    -----> Starting Kitchen (v1.1.1)
    -----> Verifying <default-ubuntu-1204>...
           Removing /tmp/busser/suites/serverspec
    Uploading /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb (mode=0664)
    Uploading /tmp/busser/suites/serverspec/spec_helper.rb (mode=0664)
    -----> Running serverspec test suite
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb --color --format documentation

    Cynthia's static html blog site
    No packages found matching apache2.
      should have installed apache (FAILED - 1)
      have enabled the apache service (FAILED - 2)
    httpd: unrecognized service
      be running the apache service (FAILED - 3)
      should listen on port 80 (FAILED - 4)
      should have a virtual host for cnk-blog

    Failures:

      1) Cynthia's static html blog site should have installed apache
         Failure/Error: expect(package 'apache2').to be_installed
           dpkg-query -f '${Status}' -W apache2 | grep '^install ok installed$'
           expected Package "apache2" to be installed
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:5:in `block (2 levels) in <top (required)>'

      2) Cynthia's static html blog site have enabled the apache service
         Failure/Error: expect(service 'httpd').to be_enabled
           ls /etc/rc3.d/ | grep -- '^S..httpd' || grep 'start on' /etc/init/httpd.conf
           grep: /etc/init/httpd.conf: No such file or directory

           expected Service "httpd" to be enabled
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:9:in `block (2 levels) in <top (required)>'

      3) Cynthia's static html blog site be running the apache service
         Failure/Error: expect(service 'httpd').to be_running
           ps aux | grep -w -- httpd | grep -qv grep
           expected Service "httpd" to be running
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:13:in `block (2 levels) in <top (required)>'

      4) Cynthia's static html blog site should listen on port 80
         Failure/Error: expect(port 80).to be_listening
           netstat -tunl | grep -- :80\
           expected Port "80" to be listening
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:17:in `block (2 levels) in <top (required)>'

    Finished in 0.06817 seconds
    5 examples, 4 failures

    Failed examples:

    rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:4 # Cynthia's static html blog site should have installed apache
    rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:8 # Cynthia's static html blog site have enabled the apache service
           rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:12 # Cynthia's static html blog site be running the apache service
           rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:16 # Cynthia's static html blog site should listen on port 80
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb --color --format documentation failed
    Ruby Script[/tmp/busser/gems/gems/busser-serverspec-0.2.6/lib/busser/runner_plugin/../serverspec/runner.rb /tmp/busser/suites/serverspec] exit code was 1
    >>>>>> Verify failed on instance <default-ubuntu-1204>.
    >>>>>> Please see .kitchen/logs/default-ubuntu-1204.log for more details
    >>>>>> ------Exception-------
    >>>>>> Class: Kitchen::ActionFailed
    >>>>>> Message: SSH exited (1) for command: [sh -c 'BUSSER_ROOT="/tmp/busser" GEM_HOME="/tmp/busser/gems" GEM_PATH="/tmp/busser/gems" GEM_CACHE="/tmp/busser/gems/cache" ; export BUSSER_ROOT GEM_HOME GEM_PATH GEM_CACHE; sudo -E /tmp/busser/bin/busser test']
    >>>>>> ----------------------

For Future Reference

  1. A tutorial in the form of a git repo: RallySoftware-cookbooks/chef-tutorials

  2. An excellent blog post