Flourish

Patches to rails

I've submitted many tickets & patches to rails. Here's a list:

• Chaining scopes with similar :joins causes table aliasing errors

  When performing an advanced search by chaining together named_scopes dynamically, I kept running into table aliasing issues. If more than one of my scopes joined in the profiles table, for example, MySQL would complain that the table was joined in more than once. There were many solutions to this problem, but the only way to keep these scopes completely independent of each other was to get ActiveRecord to remove the duplicate joins when combining the scopes. This patch simply combines multiple string identical joins into a single join. When two or more joins were specified in a scope, it also allows developers to write the joins in an array-of-strings, so that duplicates can be separated out and removed properly. Take a look at my article for more information.

  Status: COMMITTED

• Single query eager loading should be able to be globally disabled

  Optimizing the performance of a data heavy application, I found myself constantly trying to fight with rails to trigger the multi-query eager loading process. When you :include tables, rails attempts to determine if you are using these tables in your :conditions, :order, etc. If you are, it falls back to a single SELECT query with lots of LEFT OUTER JOINs. If not, it executes one query for each table, which is much faster. The problem comes from when rails guesses wrong. The guessing process is just regular expressions, which aren't parsing SQL fragments perfectly, combined with the fact that it didn't check for tables in your :join clauses, lead to some very bad guesses on my project. Since I never reference :included tables in my SQL fragments, I wanted to globally disable them so that it would always guess right.

  Status: WON'T FIX (community prefers to build better guessing)

• f.label should be able to target f.radio_button

  The IDs generated by the radio_button method on a FormBuilder can't be targets by the label method. Radio buttons almost always need labels, and those labels need to target the correct radio button to be functional. Unfortunately, the radio button IDs use the radio_button's value in them, and there's no way to make the label method generate a corresponding "for" attribute. I simply made it possible to pass :value => "something" to the label method and have it generate the same ID that a radio_button would.

  Status: PATCH SUBMITTED

• f.radio_button generates the wrong ID if inside fields_for with :index

  After trying to build a view solution for nested assignment using fields_for, I realized that the IDs generated by f.radio_button do not include the :index of their parent fields_for (unlike every other form helper). This is because the code to generate the ID for a radio_button is special because it includes the value at the end. I simply refactored this code to build on existing ID generating logic and add the value suffix at the end.

  Status: PATCH SUBMITTED

• association#find has redundant option merging logic with scope merging

  I'm trying to make associations chainable. Post.first.author.groups.members, for example, should give you all the people in the groups that the author of this post is in. Along the way to this goal, I noticed that the #find method of an association_collection does its own merging of find parameters like :order and :conditions. In this ticket, I'm hoping to convert #find to using @reflection.klass.scoped(something).find(*args).

  Status: TICKET SUBMITTED

Duplicate joins merge in rails 2.2

If you've ever suffered from duplicate table aliasing problems in rails, there's a new feature in 2.2 that will help some of these situations go away. This seems to come up mostly in named_scope, where you want to define a list of scopes that can work by themselves or with any other scopes. This often means having identical or similar joins in multiple scopes. Take this example:

class User
  has_one :profile
  named_scope :male, {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id",
    :conditions => "profiles.gender = 'male'" 
  }
  named_scope :recently_updated, {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id",
    :conditions => ["profiles.updated_at > ?", 1.week.ago]  
  }
  named_scope :admin, {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id AND profiles.admin = 1
              INNER JOIN emails ON emails.profile_id = profiles.id"
  }
  named_scope :with_profiles, {
    :join => :profile
  }
end

Before 2.2, calling User.male.recently_updated results in a table aliasing problem, because rails joins in the profiles table twice. Three features in 2.2 make this better:

• :join with hash/symbol syntax merges hashes when combining scope (Andrew White)
• :join now can take an array of strings (David Stevenson & Joseph Palermo)
• string identical joins are uniqued when combining scopes (David Stevenson & Joseph Palermo).

After 2.2, you can call User.male.recently_updated because there are two string identical joins combined in different scopes. You can't call User.male.admin without making some modifications, because the two joins involved are not string identical. Here's how I'd modify the :admin scope:

named_scope :admin, {
  :join => ["INNER JOIN profiles ON profiles.user_id = users.id",
            "INNER JOIN emails ON emails.profile_id = profiles.id"],
  :conditions => "profiles.admin = 1"
}

By using the array of strings :join syntax, I can let rails know that those are two separate joins, each of which can be merged if an identical join comes up in another scope. I also moved the extra join condition (profiles.admin = 1)to :conditions, so that the INNER JOIN statement would not have any specific logic for that particular named_scope in it.

I still can't call User.male.with_profiles because the string representation of the join from with_profiles probably doesn't match the join from male. It will be off due to the way rails generates join using different whitespace and escape characters than I originally wrote. This can be easily fixed by copying the exact string rails generates from :join => :profileand pasting it into male.

acts_as_solr has a new home

Developers finally set up master repo for acts_as_solr

After a long time of no development, a new acts_as_solr plugin has emerged on GitHub. Luke Francl & Mathias Meyer have adding a lot to the old plugin and created a new "master". Mathias has merged several branches on github including changes from David Palm, kengruven, and myself. This is great news for people running/modifying acts_as_solr, as Mathias is a real person who will actually respond to your pull requests. Some new features include:

• SOLR 1.3 installed
• Ruby 1.9 support
• LibXML 0.7+ support
• Store types as string not text fields in the index
• JVM options in solr.yml
• Completely rewritten test suite
• New solr:reindex task that automatically finds and indexes your solr models

QueryReviewer now aggregates identical queries

I've been using query reviewer exensively at pivotal, and we've noticed that it is ridiculously slow when you have a lot of queries. This is due to the large number of partials rendered (5 partials) for each query in the reviewer box.

A solution to this problem, that also brings a huge advantage, was to report aggregate statistics for near-identical queries. Similar to way mysqlslowdump groups near-identical queries, query_reviewer now also reports only 1 line for X identical queries (identical means the same structure and same stack trace). Consequently, it's a lot easier to find places where you're missing includes . Here's what it looks like now:

The time reported for a group is the total time of all the queries in that group. In addition, I add up all the times for all the queries and report that number in the header. From this, I've noticed that a lot of time is spent in rails outside of waiting for mysql to return result. I wonder where that time goes...

query_reviewer can be downloaded from github!

QueryReviewer is now rails 2.1 safe

Thanks to a nice suggestion by the community which I pretty much used verbatim. Still works with rails 2.0 and 1.2.3. Check out the project homepage.

http://code.google.com/p/query-reviewer/

I'm working at Pivotal

MySQL Query Reviewer - now with AJAX and Profiling

Kevin Hall and I have released a new version of the query_reviewer plugin. You should start by looking at my first post, to see what the basic premise is before reading this article. The single largest improvment is the ability to analyze the database requests of AJAX requests, which is accomplished by piggy backing javascript or HTML into the ends of AJAX responses. Here's what it currently looks like, analyzing my project on a page that does lots of database requests:

The improvements are:

• View query analysis for AJAX requests
• Take into account the duration of a query (if production data)
• Show PROFILE information from mysql
• Warnings for long key lengths (which can be bad even when you hit an index)

Lots of information is now available for a single slow query:

I'm really excited that this plugin is getting attention, and welcome feedback, suggestions, and bug reports on the Google Code project homepage.

To install it, simply visit the project homepage.

Flourish is back from the dead

Things started going very badly for flouri.sh when my home server wouldn't boot up. I removed almost all the hardware, but could not even get a POST screen or any beeps. I even reseated the RAM and CPU, but no luck. Finally, I grabbed an old media PC shuttle cube and attempted the craziest idea:

The 4 x RAID5 disks are actually being powered by my old server, but the PCI raid card is inserted into my old media PC. The shuttle doesn't have enough power or room to run all those disks so I'm using the old server as a lifeboat with 4 IDE cables running between them!

Writing data migrations in rails

Using ActiveRecord can be tricky, here's how

Data migrations are a major headache in every rails project I've worked on. Developers typically write straight SQL migrations which take much longer to create and test, or they use ActiveRecord and run into problems. I recent wrote a data migration which took 7 existing tables and compressed them into 4. Some of the new 4 had the same names as the existing models, so I finally figured out how to do this safely with ActiveRecord. Here's my advice:

• When changing the schema of an existing set of tables, create new tables and then rename them.
• Include active record fragments inside the migration class at the top of your migrations.
• Put your data migrations inside transactions.
• Make your data migrations completely reversible (which is a lot easier when you follow the first rule)

Here's an example of an ActiveRecord class fragment at the top of a migration:

class MarkPrimaryBits < ActiveRecord::Migration
  class LessonVersion < ActiveRecord::Base
    belongs_to :lesson
  end

  class Lesson < ActiveRecord::Base
    has_many :lesson_versions
  end
 
  def self.up
    ...

By putting the Lesson and LessonVersion classes at the top of the migration, I'm allowing myself to use those classes inside my migration and be completely independent of any changes made to the real model from then on (including the deletion of the class itself). Furthermore, I can have another migration which uses those same class names with completely different meanings and they won't conflict with each other.