CNK's Blog

Notes for Boulder RailsBridge Workshop

I am going to be teaching at a RailsBridge workshop in 2 weeks and thought I should work through the course at least once before I stand up in front of an audience! I’ll need to work through the install instructions for some other operating systems (at least the Windows instructions since that is where I have the least experience) but thought I would start by making sure my new Mac conformed to their standard setup. I had already set up Xcode, RVM, and Rails as described in a previous blog post.

So I reviewed the install instructions. Things have gotten pretty hairy on the Mac side but mostly looks OK - except they didn’t have anything about the Xcode command line tools killing terminfo files.

Since RailsBridge recommends KomodoEdit, I thought I should try it. One thing to note, it offers you some built-in Rails help - but all of that is for Rails 2 and does not seem to have been updated for Rails 3. Perhaps we should note that in our install or use instructions. I am really tempted to cheat and just tell KomodoEdit to use emacs key bindings - but that would defeat the purpose of my using the same text editor as our beginning students. So I needed some instructions on how to use it. Found these screencasts but am not yet sure how helpful they are going to be.

Outline

OK so time to start working through the curriculum.

  1. In the Ruby/irb section, make sure to emphasize that irb will echo the output from the last command. Make sure you know if the command you just ran changed the variable you are working with - or just handed you back a new value that is NOT associated with the variable, e.g. talk about the difference between flatten and flatten!.

  2. Discuss how web sites work, requests & responses.

  3. Generate topics scaffold. Walk through it in general terms - database, routes, controllers, views.

  4. Play with generated code in the browser. Suggest adding quiet_assets and thin gems in development block so log output is more sensible.

  5. Add votes resource.
    • Resource is like a scaffold but without the full CRUD support.
    • Add associations. Now a good time to play with “rails console”?
  6. Add button. Then go use it. Ooops, error message. Talk about how to read messages. Mention that Rails shows informative error messages in development mode - and opaque ones in production mode.

  7. Add controller. Discuss each part. Find, build, save!, redirect.

  8. Oooops again. I didn’t read carefully and missed the line about showing the votes. Itterate towards it: display topic.votes, topic.votes.size, add ‘votes’ - but “1 votes” looks dorky, use pluralize.

  9. Commit and Push!

  10. Rearrange page flow after adding new topics. While we are in the application layout page, mention we could spruce things up a bit. Also mention the csrf_meta_tags - built in rails security feature. If we are running ahead of schedule, mention the .json format and AJAX

  11. Simplify topic index page. Discuss link_to. Move delete links to topic details page. Discuss instance variables and loop variables. And how about displaying votes on the show page?

Extras

  1. We didn’t discuss validations any where. Add that for the topics - need a title and description.

  2. Mention the generated tests?

  3. Order topics by vote count.
    • First you need to know the difference between inner and outer joins. And then, the ActiveRecord syntax for each. If you just say “Topic.joins(:votes)”, you get an inner join - so you can never vote for a topic that doesn’t already have a vote. Dho!
      Topic.joins(:votes).group(:topic_id).count(:id)
      SELECT COUNT("topics"."id") AS count_id, topic_id AS topic_id
      FROM "topics" INNER JOIN "votes" ON "votes"."topic_id" = "topics"."id"
      GROUP BY topic_id
       => {1=>2, 2=>5}  
    • The query we want is:
      select topics.*, count(votes.id)
        from topics left outer join votes on topics.id = votes.topic_id
       group by topics.id, topics.title, topics.description; 

      To get that from ActiveRecord, we need to use some SQL fragments along with our other ARel methods - in part so that we can explicitly name the column we want to order by.

      Topic.select("topics.*, COUNT(votes.id) AS count_votes")
      .joins("LEFT OUTER JOIN votes ON topics.id = votes.topic_id")
      .group("topics.id")
      .order("count_votes DESC")
    • Another option is to add a counter cache to Topics and then increment_counter on the topic after each vote is saved (discuss ActiveRecord callbacks). Another option is to just move the votes directly into Topics. Should we? why or why not? Discuss.
  4. Making this site look less ugly. Any front end folks? Talk about the asset pipeline.

Diagrams I need:

  1. How a web site works. Click link in browser -> request -> server -> file or script -> response -> renders in your browser

  2. Our development cycle. Decide what we want -> generate/write code -> play with it locally -> git commit -> push to heroku

Things to suggest for the install notes:

  1. The problem with Xcode command tools and terminfo. Find my backup of the terminfo files and post on this site.

  2. Since we are planning to tell folks about ri in class, add the “complile docs” step to the RVM notes: $ rvm docs generate

  3. How about mentioning how to look at your SQLite3 tables in a GUI? A super simple option being: https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/

  4. What about using RailsInstaller on Mac? Or can we avoid the Xcode mess if we use rubyenv? That helps with Ruby, but what about git? and sqlite3? We can get .dmg files for git. Database? We could use MySQL .dmgs if we like.

  5. Typo in http://curriculum.railsbridge.org/curriculum/CRUD_with_scaffolding?back=creating_a_migration%23step2 Second bullet point under model belongs under view. Add a different line explaining attr_accessible - with warning that many books will not tell you to add fields to that list but you may need to.

Rspec output not colorized in emacs

Now that my machine is at least partially set up, I thought I would actually pull down some code from Github and do some programming. I have git, RVM, and sqlite3, so setting up a working environment took ~5 minutes. But there has been another security update since I last touched this code, so I thought I should start by updating everything to Rails 3.2.6 and checking to see that nothing breaks.

Well two things are broken. I now have a number of tests that are not passing. But I’ll have to get back to that later. More importantly, the colorization in my shell within emacs is broken. No pretty green (or red) in my rspec output. Not acceptable.

So first, what is the name for the system that takes the weird ^[[32mAnd "^[[32m^[[1mGe characters and turns them into pretty colors? Apparently that is called Ansi colors: http://en.wikipedia.org/wiki/ANSI_escape_code. On my Linux machines, the emacsen have ansi color interpretation turned on by default. In fact, the docs I can find for the ansi-color eLisp code indicate that colorization has been the default since ~1999. But whatever. Folks on the emacs IRC channel had a number of suggestions about how to get that working (including suggesting not using ansi-color.el but using xterm-color.el instead). The incantation that finally worked on the emacs that shipped with Lion (emacs 22.1.1 ©2007) is:

     (ansi-color-for-comint-mode-on)

While I was customizing my settings, I copied some configurations from some of my other .emacs files into here. I wasn’t sure they were compatible with the installed emacs 22 (I run emacs 23.3.1 on my server) so I tried out the automatic mode setting for Rakefiles. But, to my horror, there isn’t a ruby-mode installed here. Hmmm guess it is time to explore installing a more recent emacs.

Emacs from Homebrew

Braumeister shows an emacs 24.1 formula. Let’s see what that is like.

     ==> Downloading http://ftp.gnu.org/pub/gnu/emacs/emacs-24.1.tar.bz2
     ==> ./configure --prefix=/usr/local/Cellar/emacs/24.1 --without-dbus --enable-locallisppath=/usr/local/share/emacs/site-
     ==> make
     ==> make install
     ==> Summary
     /usr/local/Cellar/emacs/24.1: 3880 files, 102M, built in 2.1 minutes

Ahhhh much better! I have ansi color by default. And more importantly, I have ruby-mode AND my hook for automatically invoking ruby-mode for .rb, .rake, and .spec files works!

And one small tweak for this blog. The .markdown files for this blog have been opening in Fundamental mode without auto-fill on. I think text mode with auto-fill on would be more sensible:

     (add-to-list 'auto-mode-alist '("\\.markdown$" . text-mode))
     (add-hook 'text-mode-hook 'turn-on-auto-fill)

Ok now am I set up to code?

Setting Up My Lion Laptop

In the past 10 years and 2 Macs, I accumulated a lot of stuff that needs to be transferred to my new laptop. So I took advantage of the Migration option that comes up during the first-boot system setup that all Macs have. So I booted my desktop machine into transfer disk mode by restarting while holding down the ‘t’ key, then connected both machines using a Firewire 800 cable. I transferred my user account stuff, the systems Applications, and settings (total: ~240 G). The initial estimate for the transfer was 23 hours but fortunately, once it had been running for a while, it revised the estimate to ~3.5 hours. Once the initial data transfer had completed, I started clearing out stuff I don’t want or won’t work on the new machine.

I had not chosen to transfer what the Setup Assistant referred to as “Other files”. Not sure what all that encompases, but for one thing, it did not transfer the /opt directory containing my MacPorts stuff. To complete the the removal of the MacPorts stuff, I cleared out environment stuff for /opt and my Oracle vars. The Oracle instant client doesn’t run on Lion, so I removed /Library/Oracle. I also used the uninstall section of the MacPorts docs to find out what to ‘rm -rf’ in various Library directories.

The next thing I needed was a compiler, so I installed Xcode 4.3.3 from the App Store. Even with Xcode installed, I still don’t have gcc?! Apparently now one needs to ask Xcode to install an optional command line tools extension. That can be found under Preferences -> Downloads. Unfortunately there appears to be a bug in some part of the Xcode install because after I installed it, my terminal, which had been working just fine, could not run emacs:

    $ emacs
    emacs: Cannot open terminfo database file

I also see a possibly related issue: I can’t use man pages from the command line:

    $ man nmap
    WARNING: terminal is not fully functional

I can’t live with that! Sounds like the fix is to copy in a known good usr/share/terminfo directory. But where can I get one? At the moment, the fastest option appeared to be to reinstall Lion and start again.

Reinstall Lion

First I created a Lion recovery thumb drive using the instructions at http://support.apple.com/kb/HT4848 and the Lion Recovery Disk Assistant v1.0 downloaded from http://support.apple.com/kb/DL1433. Made a 2 G partition on the pink thumb drive (make sure partition table type is GUID under options) Then launched the LRDA and selected the LIONRECOVER partition to receive the data. The recovery stuff copied over in < 15 min and I was able to eject the disk.

To do the actual recovery, I needed to be connected to the internet. Wired network is the fastest - esp as I went to work and used their fast network. I restarted my laptop while holding down Command+R. Because this restart is before my settings kick in I had to use the real Command key - with the apple symbol on it. The laptop booted into recovery mode, checked with Apple to make sure this was a machine that shipped with Lion on it, and then reinstalled. The reinstall took about an hour and left all my user data intact and most of my applications.

Once it had rebooted, I had a nice complete set of /usr/share/terminfo and emacs was happy once more. I made a backup and then checked Xcode. Xcode was still installed in the Applications folder but when I started it, it went through the same ‘first time’ operations as it had the first time. When I looked at the Preferences -> Downloads the 2 iPhone simulator add ons that I had installed the first time were still installed. But the Command Line Tools showed they were not. I reinstalled them and then checked my term info by trying to run emacs again. Nope, same error message about my terminfo database. So clearly the root of the problem is with installing the Xcode command line tools. I untarred my backup terminfo directory and replaced the broken one with the original information from my clean install.

Homebrew

I already had some stuff installed in /usr/local. I don’t think I want most of it, but was concerned that just removing the directory contents would not cleanly uninstall everything. So for my first attempt, I followed the install instructions and ran the install script as:

    /usr/bin/ruby -e "$(/usr/bin/curl -fsSL https://raw.github.com/mxcl/homebrew/master/Library/Contributions/install_homebrew.rb)"

Once that had run, it told me to run “brew doctor”. That gave me a lot of warnings about the things that were already installed. So I decided to remove things that were installed by homebrew (.gitignore, Cellar, Library, README.md), tarred up the rest, and cleared everything out of /usr/local. This time after I installed homebrew and ran “brew doctor” I got “system is raring to brew”.

So how do I figure out what formulas are available? One option is from the command line: brew search mysql. That will list all available formulas that have mysql in their name. Another option is searching on http://braumeister.org/

To find out what I have installed, I can look in /usr/local/Cellar/. To find out what version of a package is currently active, I can look at the symlinks in the bin or lib directories. Or I can use the command line: brew list wget; that will list all the files associated with that package.

I may want to override some system packages with ones of my own, so in my .profile, I added /usr/local/bin into path before /usr/bin.

MySQL

Lion has SQLite3 and Postgresql 9 already installed, but I don’t see any sign of MySQL so I installed MySQL 5.5 from brew. After installing, I get a bunch of instructions about creating databases and starting when I boot/login. So first of all, I need to install the main database:

    mysql_install_db --verbose --user=`whoami` --basedir="$(brew --prefix mysql)" \
                     --datadir=/usr/local/var/mysql --tmpdir=/tmp

The output from mysql_install_db gave me the familiar “be sure to set a root password” warning. It is less clear that you have to start mysql.server first and then use the mysql_secure_installation script.

    mysql.server start
    /usr/local/Cellar/mysql/5.5.25/bin/mysql_secure_installation
    # I set a root password, disabled remote root access,
    # and got rid of the test databases and access

To make the mysql server run when I log in, I followed the instructions that homebrew gave me upon install:

    cp /usr/local/Cellar/mysql/5.5.25/homebrew.mxcl.mysql.plist ~/Library/LaunchAgents/
    launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist

RVM

Lion comes with Ruby 1.8.7 installed and the gems from my previous Mac appear to have transferred. But for more recent rubies, I’ll want to use RVM. I had installed RVM on my previous Mac but haven’t really used it, so I think I’ll start over. First I removed the old RVM with “rvm implode”. Then I followed instructions at https://rvm.io/rvm/install/

    curl -L https://get.rvm.io | bash -s stable --ruby

This installed ruby 1.9.2-p194, gem, bundler, rvm, and rake. Then to do a basic rails install:

    rvm gemset create rails-3-2-6
    rvm gem set use rials-3-2-6
    gem install rails mysql pg sqlite3

Setting up Octopress

I have had a web site forever - but not really a blog. Guess it is time to experiment with the date-driven style and see what I think. At least with Octopress (or Jekyll), I don’t have to give up my beloved emacs + git for an unwieldy CMS. (Confession: other than some tedium remembering the not-quite-html tag syntax, I really love DocBook for collecting my notes, how-tos, etc.)

Following the instructions on the Octopress setup page, I cloned the repository from Github. When I cd’ed into the directory, RVM asked me if I would approve the .rvmrc file that required Ruby 1.9.2. I am trying to move all my development to Ruby 1.9.3 so I would prefer not to install 1.9.2 if I don’t have to. I looked around (esp. through the issues queue) to try to see why the requirement is for 1.9.2 instead of just > 1.9.1. All I could find was a ticket that said “just use RVM and install 1.9.2 already!” but didn’t enumerate any issues with 1.9.3. So I may end up falling back but at least for the initial setup, I am going to push ahead with 1.9.2-p194 which is the current stable patch level.

    rvm gemset create octopress
    bundle install
    rake install

All seems to have installed and run just fine - even the dependencies like posix-spawn, fast-stemmer, ffi, and rdiscount which have native extensions. So I updated my copy of the .rvmrc file with my ruby and gemset.

Next I followed the instructions for setting up rsync deployment - putting my information in the Rakefile. And configured a few options to add my GitHub and Twitter feeds to show in the sidebar. (Still need to figure out why the GitHub section still says “Status updating…”) and then used the supplied rake task to create the file for this post. The preview (rake preview) looks great!

It remains to be seen if I am going to like markdown. Having to close tags is tedious, but in the end, it is often less time consuming than looking up the alternative syntax for the non-html markups. But looks like I can use at least some regular html tags inside this markdown file if I want to.

SQLAlchemy from an ActiveRecord Perspective

I have always started any project from the database perspective. The whole test-first/behavior-driven-development movement has led me to think I would be better off starting from the user interface and throug the stack to the back end. But for me the data model layer is always the easiest. And I actually kind of like SQL so the ORM claims that “you’ll never have to write SQL again” were not a strong selling point - at least not for me. For me the biggest selling point for ActiveRecord (the first ORM I really used) was that I didn’t have to write all the getters/setters in the Ruby side. That and the ease of grabbing related objects and getting them as real objects - not as rows containing attributes from both tables. All the Ruby metaprogramming magic that gives you Foo.find_by_name is nice and makes for very readable code, but that is merely a convenience for me, not a necessity like the object creation feature.

My group at work is in the process of moving from a very simple home grown…. I am not sure it is actually a full blown ORM… perhaps automated query builder? We are a python shop and have decided to adopt SQLAlchemy (currently version 0.7.2). I am in the process of reading the tutorial for the first time and wanted to record my initial impressions.

For starters, as you might expect from the difference in Ruby vs. Python cultures, SQLAlchemy is more explicit than ActiveRecord. SQLAlchemy has what is called a “declarative” mode where you define your class and your table all at once (within a class that inherits from SQLAlchemy’s declarative_base class). But you can also explicitly create, edit, and inspect an explicit metadata object which defines the table. Then you can define a Python class for your object - and then use SQLAlchemy’s mapper to introduce the two.

Non-database attributes

How do the two ORMs deal with model attributes that do not have corresponding database columns? For example, user models commonly ask for a password and a password confirmation in a user form. But what is generally stored in the database is an encrypted form of the password (sometimes called encrypted_password). In Rails/ActiveRecord, you generally create the database migration with the columns that should be in the database, then add password and password_confirmation attributes to the model using attr_accessible. And then you create a "before_save" filter that encrypts the password and stores it in the encrypted_password field. In SQLAlchemy, you can create tables and model classes with different attributes. The mapper step matches the attributes with the same names AND appears to have added an "encrypted_password" attribute to the python object - even though it was not defined in the python class.

Commit behavior

With SQLAlchemy it seems you always have to explicitly commit to get changes written to the database. I have gotten used to ActiveRecord’s mix of implicit and explicit writing to the database. ClassName.create() automatically saves - but ClassName.new() does not. Updates to an ActiveRecord object usually need to be explicitly saved to write to the database. There are some exceptions, for example, when you assign an object to a has_one association, the parent object is automatically saved (in order to update its foreign key). Reading the ActiveRecord docs it sometimes sounds confusing, but in practice, it usually behaves as I want it to. On the other hand, SQLAlchemy’s requirement for an explicit session.commit() also means it is easier to intervene with an explicit session.rollback() if you decide you don’t want to persist the changes you have made to your Python object.

Data Definition

SQLAlchemy has nice, explicit syntax for defining your schema but still in a database agnostic way. Column('name', String(30)) is a pretty easy mental mapping from name varchar(30). The ActiveRecord equivalent, t.string :name, :limit => 30, isn’t bad but isn’t superior either.

Query syntax

The new Rails3 syntax - with chained method calls rather than a hash of options - looks more like SQLAlchemy than it used to. And I suspect they may behave more similarly too - building up a query from pieces and then executing it. But just the fact that the SQLAlchemy tutorial covers how to use subqueries shows that writing SQLAlchemy queries is closer to real SQL than writing ActiveRecord is.

If you have a particularly tricky set of SQL, both ORMs let you create your own SQL. In ActiveRecord this is done with the find_by_sql method. In SQLAlchemy, it is done using from_statement:

    session.query(User).from_statement("SELECT * FROM users where name=:name").params(name='ed').all()

SQLAlchemy can return an iterator or an object or a list. session.query(PythonClassName).filter(<conditions) returns an iterator that will feed you instances of PythonClassName in a loop. You can also return objects and list of objects by setting the query to session.query and then calling query.one(), query.first(), or query.all(). The query.one() behaves rather like ActiveRecord’s ClassName.find()</code> in that it wants exactly one result row and will throw and exception if the return is not exactly one row.

Relationships

Both ORMs have syntax for defining relationships. The ActiveRecord syntax is very English-like: has_one :foo, belongs_to :bar, has_many :widgets. The SQLAlchemy syntax is not as streamlined

  • but it may make it easier to do some explicit joining. ActiveRecord is fabulous - unless your schema doesn’t want to use its naming conventions. There are, in theory, modifiers to the relationship defining methods that are supposed to let you get around that. But in practice, at least as of Rails 2.3, they don’t always work consistently and you can get a ways into your code before discovering, for example, that the cascading delete of related objects doesn’t work because the generated sql is looking for the column “id” rather than “foo_id”. In SQLAlchemy the syntax is:
    class User(Base):
        addresses = relationship(Address, order_by=Address.id, backref='user')

    class Address(Base):
        user = relationship(User, backref=backref('addresses', order_by=id))

Many to many relationships are supported in both ORMs and both make a distinction between m-to-m relationships with boring association tables (which only contain the 2 foreign key columns) and richer associations where you store additional data on the association (for example audit information about when it was created, and by whom). In ActiveRecord, those two cases are :has_and_belongs_to_many and :has_many_through. In SQLAlchemy, the simple case is taken care of by adding "secondary="</code> to the relationship definition. The richer case is taken care of by creating an association object and then establishing relationships between each object and the association object. The example in the association documentation isn't super clear (I think calling the relationships left and right make the names look like keywords). But [this example] (http://www.preetk.com/node/sqlalchemy-part-2-declarative-bi-directional-association-classes/) using a standard permission setup makes better sense.

Both ORMs support lazy loading of related objects by default but can be asked to do eager loading if you ask it to using joinedload() (SQLAlchemy) or .include() (ActiveRecord)

Web form building

Rails’ ActiveSupport provides form building helpers to take the tedium out of building data entry forms. The equivalent in the Python/SQLAlchemy world is provided by FormAlchemy. I can’t say I really like the way FormAlchemy does data validation. I found it rather difficult to figure out how to do custom validations - especially those where the requirements for one field depend on the value in another field. In ActiveRecord this is straightforward.