CNK's Blog

Getting started with AppFog

This week’s Ruby Weekly was sponsored by AppFog. It sounded interesting so I thought I would give it a whirl. Their pricing model is interesting - it is based on how much RAM you use (free if you use less than 2 GB!). Seems sensible to me since that is what I have usually found to be the limiting factor for the kinds of sites I run. And I’ll have to say that their “Jumpstart” setup couldn’t have been easier - two clicks and a name for the app and I had a Rails app running in the cloud in well under 2 minutes. Of course all it does is show the “Welcome to Rails” page we all know and love. So now how do I make this useful?

If I were not a document-reading sort of person, it would have taken me less time to figure things out. When you are logged in, your AppFog console page is a list of your apps. Clicking on the app name gets you to “Mission Control” for the app and one of the tabs on that interface is “Update Source Code”. That page provides the details about how to install the ‘af’ command-line tool that were missing from the AF CLI doc page I had looked at first. It confirmed my suspicions that ‘af’ was distributed as a gem (even though all the docs did was link to the GitHub repository). I will probably want to use this for more than one project, so I’ll want ‘af’ available in all my gemsets. So I did:

    $ rvm gemset use global
    $ gem install af
    Successfully installed json_pure-1.6.7
    Successfully installed rubyzip-0.9.9
    Successfully installed mime-types-1.19
    Successfully installed rest-client-1.6.7
    Successfully installed terminal-table-1.4.5
    Successfully installed interact-0.4.6
    Successfully installed addressable-2.2.8
    Successfully installed uuidtools-2.1.3
    Successfully installed rb-readline-0.4.2
    Successfully installed af-0.3.18.2

Once past that hurdle, things look better:

    $ af login
    $ af apps

    +-------------+----+---------+--------------+-----------------+----+
    | Application | #  | Health  | URLS         | Services        | In |
    +-------------+----+---------+--------------+-----------------+----+
    | cnk         | 1  | RUNNING | cnk.rs.af.cm | cnk-mysql-29232 | rs |
    +-------------+----+---------+--------------+-----------------+----+

OK now for that documentation on actually using the tool.

I actually ended up using a combination of methods. I used the “Download Source Code” button on the web interface to get a copy of the generated code that is on my running site. I swapped out the “Welcome to Rails” page for my boring test message and then used ‘af’ to redeploy the new code:

    $ af update cnk --path appfog-test
    af update cnk --path appfog-test
    Uploading Application:
      Checking for available resources: OK
      Packing application: OK
      Uploading (38K): OK
    Push Status: OK
    Stopping Application 'cnk': OK
    Staging Application 'cnk': OK
    Starting Application 'cnk': OK

One odd thing, I couldn’t run any ‘af’ commands if I was in the directory of Rails code I downloaded from AppFog. Any ‘af’ subcommand I tried errors with:

    $ af apps
    af apps
    /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/gems/bundler-1.1.4/lib/bundler/spec_set.rb:90:in `block in materialize:
    Could not find multi_json-1.3.6 in any of the sources (Bundler::GemNotFound)
          from /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/gems/bundler-1.1.4/lib/bundler/spec_set.rb:83:in `map!'
          from /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/gems/bundler-1.1.4/lib/bundler/spec_set.rb:83:in `materialize'
          from /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/gems/bundler-1.1.4/lib/bundler/definition.rb:127:in `specs'
          from /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/gems/bundler-1.1.4/lib/bundler/environment.rb:27:in `specs'
          from /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/gems/rubygems-bundler-1.0.3/lib/rubygems-bundler/noexec.rb:41: in `candidate?'
          from /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/gems/rubygems-bundler-1.0.3/lib/rubygems-bundler/noexec.rb:60:in `setup'
          from /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/gems/rubygems-bundler-1.0.3/lib/rubygems-bundler/noexec.rb:75:in `<top (required)>'
          from /Users/cnk/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:60:in `require'
          from /Users/cnk/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:60:in `rescue in require'
          from /Users/cnk/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:35:in `require'
          from /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/bin/ruby_noexec_wrapper:9:in `<main>'

I think for now I’ll ignore that problem.

Overall, this is vary clean and neat. One thing I already miss just a little is Heroku’s git-based deploy strategy. Being the version control junkie I am, there is something so right and satisfying in deploying with ‘git push heroku master’. So can you use git to deploy to AppFog? Apparently yes. It is a little more involved - you have to run a Node.js app that you can use as the url for a git post-receive hook. That let’s you automatically deploy whenever you push to the prodution (or whatever) branch.

Heroku also has some very convenient tools for moving data to and from your production database (using the taps gem). The equivalent thing in AppFog appears to involve ssh-tunnelling - see this post for details.

So far AppFog is looking promising. For most of what I do Heroke is amazingly convenient - and it now supports stacks other than Ruby on Rails. But one thing I noticed listed in the AppFog Jumpstart lists is Drupal. I am not sure if Heroku supports Drupal or not - but if I need a Drupal site and need more control than one can get hosting at Drupal Gardens, AppFog might be an option.

Upgrading from Rails 2.3.5 Part 2

csrf_meta_tags

Carrying on from part1, once I had everything in place to run my tests, almost all of my functional tests failed complaining “no method csrf_meta_tags”. Ahh right, I am not running the stock Rails 2.3.5 gem in production. In February of 2011 the rails team put out a patch for a XSS vulnerability. I was concerned that some of the changes between Rails 2.3.5 and 2.3.10 might interfere with my connection to my Oracle 8i database. I was already having to do some monkey patching of the activerecord-oracle_enhanced-adapter so I could continue to use it. So instead of upgrading my gems to Rails 2.3.11, I applied the patch and then added <%= csrf_meta_tags %> to all of my layouts. Since the point of what I am doing is to upgrade my gems, for now I just commented out the offending line in each of my layouts.

SSL

OK now that I actually run some functional tests, there are a couple of other things that are not working properly. On my production (and usual development) machines, I have ssl configured and my application uses the ssl-requirement plugin to require my login pages are only accessible under ssl. I don’t want to set up SSL on this temporary location, so for now I tweaked the plugin to claim that all protocols are ‘secure’. Also during a security audit, I was asked to only pass my user cookie over https to prevent session hijacking. Since I disabled SSL, then I can never login; so I had to temporarily disable the secure cookie requirement too. Interestingly a recent Railscast discussed session hihacking and provided example code basically like what I have.

Mime-type checking

The next set of failures in my functional tests are in tests that check to see that I can only uplaod specific file types into the image and documents sections of my CMS. By default my attachement processing code, which uses attachment_fu, was using the content type provided during file upload - which largely uses the file extension to determine file type. To beef that up, I have extended attachment_fu to use the shared-mime-info ruby gem. It works fine on my production RHEL5 machine and as far as I can tell I have the same rpm + gem combination on this test box. But it isn’t working. All of my file uploads (in tests and from the browser) are getting back nil for the mime-type. Looking at the code, that implies that the library thinks it should be able to determine the file type but can’t. This may be that I am not loading the library correctly in my Gemfile: gem "shared-mime-info", :require => "shared-mime-info" But I have already had trouble getting this to work on my collegue’s Leopard Mac so there is already a work around in place for that. Again, for now I am going to make this work around trigger all the time and see what else I need to change to get this app on a supported version of Rails.

“Real” errors

So after ignoring/working around the issues above, are all my tests and cucumber features passing? Yes! I am getting one warning when I run my cucumber features. I am being told that I need to update one of my steps that is reusing other existing steps. I have been using the original syntax that uses ‘When/Given/Then’. The warning tells me to use ‘step’ instead. I made that change and then moved on.

Rails 2.3.14 on Ruby 1.8.7

So, what happens when I update to the last of the 2.3.x releases - 2.3.14 at this time. I created a new rvm gemset, updated the rails line in my Gemfile, removed my Gemfile.lock, and did a clean bundle install. When I first tried to run my app, I got complaints that I did not have Rails 2.3.5 installed - which was coming from my config/environment.rb file:

     # Specifies gem version of Rails to use when vendor/rails is not present
     RAILS_GEM_VERSION = '2.3.5' unless defined? RAILS_GEM_VERSION

So now let’s run our tests again. They pass, but this time with some additional deprecation warnings. The first set were complaints about the preferred location of the tasks directory inside my plugins. Errors like:

    DEPRECATION WARNING: Rake tasks in vendor/plugins/aftfilr/tasks are deprecated. Use lib/tasks instead.
    (called from /home/cnk/.rvm/gems/ruby-1.8.7-p352@rails2.3.14/gems/rails-2.3.14/lib/tasks/rails.rb:10)

None of the directories that rake was complaining about actually contained any tasks so the simplest option would seem to be to remove the offending directories. Or it would be if they were in code I had written. But all of those plugins were code I was including using git submodules and pointing directly to the upstream repositories. I suspect that none of those plugins have been updated in a very long time. Perhaps the best/easiest option is to just include those plugins directly in my code. Which plugins do I need to alter: aftflr, aftimagr, authorization, and mimetype_fu. Of the 4, the only one that has had any updates is mimetype_fu. Mimetype_fu is used when I have to work around the lack of shared-mime-info code on the Mac (or on this test box). For now I think I’ll just include this in my code as is.

    NOTE: Gem.source_index is deprecated, use Specification. It will be removed on or after 2011-11-01.
    Gem.source_index called from
    /home/cnk/.rvm/gems/ruby-1.8.7-p352@rails2.3.14/gems/rails-2.3.14/lib/rails/gem_dependency.rb:21.

    DEPRECATION WARNING: Kernel#returning has been deprecated in favor of Object#tap.
    (called from enable_activerecord at /home/cnk/rails/empcms2.3.5/vendor/plugins/will_paginate/lib/will_paginate.rb:39)

    DEPRECATION WARNING: Kernel#returning has been deprecated in favor of Object#tap.
    (called from generate_img_css_classes at /home/cnk/rails/empcms2.3.5/lib/aftimagr.rb:22)

Upgrading from Rails 2.3.5 Part 1

Upgrading old software is often painful - but it is especially painful if you wait for a long time so that there are LOTS of changes between the old and new. The main thing that had prevented me upgrading before was a direct connection from my Rails CMS to a legacy Oracle 8i database. I no longer need to do that, so perhaps it is time to catch up with modern Rails.

There are a lot of blog posts about upgrading from Rails 2 to Rails 3; one of the most useful seems to be this one from Simon Carletti that was written before Rails 3 actually came out: The Road to Rails 3: make your Rails 2.3 project more Rails 3 oriented

His recommended first step is to get your Rails 2.3 app running using Bundler to manage your dependencies. The bundler web site has a page on how to use bundler with Rails 2.3 Seems fairly straightforward so I made the adjustments in the Rails boot files and moved my “config.gem” lines from the environment files into a Gemfile. One thing that had been kind of odd in my original project is that Shoulda, my favorite testing tool, would only seem to work if I required it globally, e.g. in config/environment.rb rather than in config/environments/test.rb. Since I am trying to change only one thing at a time, I put shoulda in the global part of my Gemfile and not just in a test section.

Which ruby?

OK so now that I have a Gemfile, which ruby should I try installing it in. Ruby 1.8.7 has had its last feature release and everyone has moved on to Ruby 1.9.3, which should be backwards compatible with Rails 2.3.5. Since that is the only ruby I currently have installed in RVM on my Lion laptop, let’s start with that one. I created a gemset (and an .rvmrc file) and then did ‘bundle install’. I had to do a little fiddling with versions, etc. Some things I knew wouldn’t work with the newer version, for example rails and shoulda. Others I didn’t know but guessed that if there had been more than a couple of minor version updates since the versions I am running in production, the new versions probably wouldn’t work with my older code. So I was fairly aggressive in specifying exactly which versions to use.

Once I had a promising looking Gemfile and installed gems, I tried running the rails console - and then one of the stock rake tasks. Sadly both failed. The script/console error is:

    (brazen:~/rails/empcms2.3.5) $ script/console
    script/console
    /Users/cnk/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in
    `require': cannot load such file -- script/../config/boot (LoadError)
            from
            /Users/cnk/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in
            `require'
            from script/console:2:in `<main>'

And the rake error is:

    (brazen:~/rails/empcms2.3.5) $ rake db:schema:dump
    rake db:schema:dump
    WARNING: 'require 'rake/rdoctask'' is deprecated.  Please use 'require 'rdoc/task' (in RDoc 2.4.2+)' instead.
        at
        /Users/cnk/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/rdoctask.rb
    rake aborted!
    undefined local variable or method `version_requirements' for
    #<Rails::GemDependency:0x007fb19601a4b0>

    Tasks: TOP => db:schema:dump => environment
    (See full trace by running task with --trace)

The rdoc warning is harmless (though annoying) - the alumni code running 2.3.14 is running fine despite similar warnings - but the others are showstopping.

There are some differences between where Ruby 1.8.7 and 1.9.3 will search for files when you ‘require’ or ‘load’ them. I am not sure if that is the cause of the problems I had, but I decided that perhaps upgrading one thing at a time might be the better part of valor. Let’s try falling back to Ruby 1.8.7.

Ruby 1.8.7

Unfortunately I can’t install ruby 1.8.7 on this Lion laptop - or at least not without a lot of work:

    $ rvm install 1.8.7
    rvm install 1.8.7
    The provided compiler '/usr/bin/gcc' is LLVM based, it is not yet
    fully supported by ruby and gems, please read `rvm re\
    quirements`.
    [11:56 PM] (brazen:~/rails/empcms2.3.5) $ rvm requirements
    rvm requirements

      Notes for Mac OS X 10.7.4, Xcode 4.3.3.

    For MacRuby: Install LLVM first.

    For JRuby:  Install the JDK. See
    http://developer.apple.com/java/download/  # Current Java version "1.6.0_26"
    For IronRuby: Install Mono >= 2.6
    For Ruby 1.9.3: Install libksba # If using Homebrew, 'brew install libksba'

    You can use & download osx-gcc-installer:
    https://github.com/kennethreitz/osx-gcc-installer ** NOTE: Currently,
    Node.js is having issues building with osx-gcc-installer. The only fix
    is to install Xcode over osx-gcc-installer.

    We had reports of http://hpc.sourceforge.net/ making things work, but
    it looks like not easiest/safest to setup.

    To use an RVM installed Ruby as default, instead of the system ruby:

        rvm install 1.8.7 # installs patch 357: closest supported version
        rvm system ; rvm gemset export system.gems ; rvm 1.8.7 ; rvm
        gemset import system.gems # migrate your gems
        rvm alias create default 1.8.7

    And reopen your terminal windows.

    Xcode 4.2:
     * is only supported by ruby 1.9.3+
     * it breaks gems with native extensions, especially DB drivers.

    Xcode 4.3+ users
    - please be warned
    - only ruby-1.9.3-p125+ is partially supported
    - in case of any compilation issues:
     * downgrade to Xcode 4.1
     * uninstall Xcode and install osx-gcc-installer
    and reinstall your rubies.

Trying to install REE gives a similar message about not liking my gcc compiler. It suggests installing “osx-gcc-installer”.

I don’t love the advice about downgrading Xcode - especially with the other note about osx-gcc-installer not playing well with Node.js. I am not currently using Node.js - but I am interested in trying it. So let’s do this on Linux.

On a RHEL5 VPS

I already have RVM and Ruby 1.8.7p352 installed on the VPS I share with friends. So I tarred up and moved the code from my Mac to that server. Altered the .rvmrc to create a gemset for ruby 1.8.7 and rails 2.3.5. I did not have bundler in that ruby so I installed it into the gemset by hand - and then did “bundle install to get the rest

    $ gem install bundler
    $ bundle install

Looks fine - but I need to configure my database. It is getting late. Take that up tomorrow.

Rails Installer on Windows XP

I hear good things about using Rails Installer on Windows but thought I should have some first hand experience before the RailsBridge install fest on Friday. So I fired up the old Windows XP box in the back room and after installing all the Windows updates, I got started. I clicked “Download the Kit” and double clicked it to run. Almost immediately I got a warning about a file that was not accessible AND a popup window from Semantec AntiVirus. So I launched the tool and under “Configure” -> “File System Auto-Protect” turned off the auto-protect while the installer was running.

In the command prompt that was supposed to configure git and ssh, I seem to already have values. It came up with user.name and user.email for me - but I am not sure where it got my name and email address. Is that configured somewhere in Windows? And I should have checked for ssh keys before I ran the installer. It has a key pair with an older date - but it isn’t my usual pair. It might be my key pair from when I was playing with PGP encrypting my mail on ugcs ages ago. But not I don’t see that pair there so I have my doubts.

The stated versions of things are: Git 1.7.9.msygit Ruby 1.9.3p125 Rails 3.2.1

Looks pretty good. Wonder if there would be any problems upgrading the Rails gems to pick up the security patches.

Rails new works and I can start up Webrick in the usual way. Semantec asks me if I want to keep blocking "Ruby" and I answered no - and then I could see the welcome page at http://localhost:3000

Github stuff

The installer creates a task called publickey that copies the generated public key to your clipboard. On XP it complained about ‘clip’ not being a recognized command - but then worked just fine. I could paste my new public key into the appropriate place in Github. I think the path changes that RailsInstaller made only work in the Command Prompt app. I could push to gethub from the terminal but NOT from the shell in emacs. In emacs, my pushes just hung and never timed out or errored. I could Ctr-C out of them - and it always asked me if I wanted to terminate a batch job. I doubt this will bother any of our RailsBridge students so I am not going to investigate any further.

The video on the RailsInstaller site discusses deploying on EngineYard. We are planning to deploy the RailsBridge stuff on free Heroku instances, but I suspect some of the hints may be of use to us. EngineYard uses MySQL instead of Postgres but the “add the database driver for group :production” is the same. The video then says to run “bundle install –without production” so that version information about the database driver gets into the Gemfile.lock. THEN the video told me to edit Gemfile.lock and delete “x86-mingw32” from the version numbering for mysql2 - and anywhere else I see it. It also said to add “ruby$” as a line under PLATFORMS (above the “x86-mingw32” line).

Installing PostgreSQL with homebrew

When I was poking around to see what was installed, I found psql in /usr/bin/psql and I had taken that to mean that I already had PostgreSQL installed. However, looking at the listening ports and in the socket directory, I don’t see any evidence of a running server. Hmmm……

  $ which psql
  /usr/bin/psql

  $ psql --version
  psql (PostgreSQL) 9.0.5
  contains support for command-line editing

  $ ps -ef | grep -i sql
  # .... only see stuff for mysql

  $ lsof -i -P | grep LISTEN
  ruby      16322  cnk   11u  IPv4 0xffffff801910cfa0      0t0  TCP    *:3000 (LISTEN)
  mysqld    29721  cnk   10u  IPv4 0xffffff8018365500      0t0  TCP    *:3306 (LISTEN)

  $ find /usr -name pg_ctl
  # Got nothing

  $ psql
  psql: could not connect to server: Permission denied
    Is the server running locally and accepting
    connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?

   $ sudo ls -al /var/pgsql_socket
  total 0
  drwxr-x---   2 _postgres  _postgres   68 Apr 26 16:40 .
  drwxr-xr-x  28 root       wheel      952 Jul  4 12:16 ..

Googling around it appears that the Lion Server may have the full PostgreSQL server running but my laptop seems to only have the psql client installed. OK let’s whip out homebrew. There appear to be formulas for versions 8 and 9 - but the default is now 9 so:

    $ brew install postgresql

That compiled and installed PostgreSQL 9.1.4. Then I followed the instructions given at the end of the install:

    If this is your first install, create a database with:
      initdb /usr/local/var/postgres

The initdb had a couple of interesting messages. First, The files belonging to this database system will be owned by user "cnk". This user must also own the server process. Ok that isn’t a problem - on a laptop anyway. And it also warned me that it was enabling "trust" authentication for local connections. The notes in /usr/local/var/postgres/pg_hba.conf warn me that local “trust” authentication means that any local user can connect as any databases user - including the database superuser. Again more permissive than I generally am, but probably OK - and certainly convenient - as long as I am only accepting connections from localhost.

The next useful piece of information from the brew install script was how to have PostgreSQL start up when I log in:

    If this is your first install, automatically load on login with:
      mkdir -p ~/Library/LaunchAgents
      cp /usr/local/Cellar/postgresql/9.1.4/homebrew.mxcl.postgresql.plist ~/Library/LaunchAgents/
      launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

Homebrew also shows it’s Ruby leanings by helpfully suggesting the compile flags I may want when installing the pg ruby gem. Since I may want this available to me in a variety of gemsets, let’s see if I can just install it in the global gemset for Ruby 1.9.3 (the only Ruby I have installed at the moment).

    $ rvm gemset use global
    $ env ARCHFLAGS="-arch x86_64" gem install pg
    Fetching: pg-0.14.0.gem (100%)
    Building native extensions.  This could take a while...
    Successfully installed pg-0.14.0

OK so now that we are installed and running, let’s log in:

  $ psql
  psql: FATAL:  database "cnk" does not exist
  $ createdb cnk
  createdb cnk
  [10:46 PM] (brazen:~) $ psql
  psql (9.1.4)

  cnk=#